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I. 


INTRODUCTION 


A.  PURPOSE 

The  Defense  Logistics  Agency  (DLA)  manages  the  distribution  (receipt,  storage 
and  issue)  of  material  across  all  components  of  the  Department  of  Defense  (DOD).  We 
researched  forecasting  models  to  more  accurately  predict  DLA’s  distribution  workload. 
This  distribution  workload  forecast  is  used  to  help  properly  staff  distribution  centers  and 
set  rates  to  fully  recover  costs.  So,  workload  forecast  accuracy  is  important  for  DLA  to 
remain  cost-competitive. 

B.  THE  PROBLEM 

DLA’s  current  forecasting  method  is  not  sufficiently  accurate  in  predicting 
workload,  and  this  degrades  their  ability  to  set  rates,  staff  distribution  centers  and  fully 
recover  costs.  Inaccurate  workload  predictions  can  contribute  to  improper  staffing 
decisions  at  the  distribution  centers  (DC),  affecting  order  processing  times. 

According  to  DLA  personnel,  the  current  regression  model  used  to  estimate  DLA 
distribution  processing  workload  has  become  an  unreliable  tool.  The  model’s 
ineffectiveness  was  accentuated  by  the  recent  budget  downturn  across  the  Department  of 
Defense. 

C.  RESEARCH  QUESTIONS 

Our  research  examined  the  current  regression  model  DLA  uses  to  forecast  its 
workload  across  its  distribution  centers.  We  identified  factors  and  issues  affecting  the 
accuracy  in  predicting  DLA  distribution  workload.  Our  research  sought  to  answer  two 
primary  questions.  First,  what  are  the  shortcomings  of  the  current  regression  model? 
Second,  are  better  forecasting  methods  or  tools  available  to  provide  more  accurate 
forecasts? 
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II.  BACKGROUND 


A.  DEFENSE  LOGISTICS  AGENCY 

The  DLA,  originally  established  as  the  Defense  Supply  Agency  (DSA)  in  1961,  is 
critical  to  providing  material  and  service  support  to  our  military  services  and  our  national 
security  objectives. 

1.  History 

Prior  to  DSA’s  establishment,  each  military  service  managed  its  own  consumable 
and  commodity  items  as  well  as  supply  processes.  The  goal  of  a  single,  consolidated 
management  agency  was  to  create  efficiencies  in  procedures  and  reduce  inventories  and 
overhead,  while  providing  timely  support  for  the  military  services  and  contingency 
operations  (Defense  Logistics  Agency  [DLA],  2011/ 

Under  the  direction  of  Secretary  of  Defense  Robert  McNamara,  DSA 
consolidated  the  following  eight  single  management  agencies  (DLA,  2011): 

•  Defense  Clothing  and  Textile  Supply  Center  (Philadelphia) 

•  Defense  Construction  Supply  Center  (Columbus) 

•  Defense  General  Supply  Center  (Richmond) 

•  Defense  Medical  Supply  Center  (Brooklyn) 

•  Defense  Petroleum  Supply  Center  (Washington,  DC) 

•  Defense  Subsistence  Supply  Center  (Chicago) 

•  Defense  Traffic  Management  Services  (Washington,  DC) 

•  Defense  Logistics  Services  Center  (Washington,  DC) 

The  administration  of  these  eight  commodity  centers  began  the  expansion  and 
increased  responsibilities  that  DSA  (renamed  DLA  in  1977)  experienced  over  the  next 
several  decades. 
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In  1990,  the  Department  of  Defense  directed  DLA  to  manage  the  unified  material 
system,  consolidating  all  the  distribution  depots  in  an  effort  to  reduce  overhead  and 
inventories.  To  achieve  this,  DLA  began  adopting  commercial  business  practices, 
automating  and  modernizing  their  depots  and  processes.  DLA  introduced  an  enterprise 
resource  planning  (ERP)  initiative  called  the  Business  Systems  Modernization  (BSM) 
program,  which  it  integrated  throughout  its  supply  centers  by  2007  (DLA,  2011). 

According  to  DLA  Loglines  (2011),  in  the  1990s,  DLA  reduced  the  number  of 
organizations  reporting  to  the  DLA  director  from  42  to  six  through  integrating  business 
units.  This  integration  continued  through  2010  as  business  units  fell  under  the  DLA 
unified  integrated  enterprise.  Base  Realignment  and  Closure  initiatives  throughout  the 
2000s  pushed  this  further,  with  the  goal  of  making  DLA  more  efficient. 

In  the  wartime  years  following  9/11,  DLA’s  business  doubled.  DLA  focused  on 
customer  support,  getting  the  right  material  at  the  right  time  to  the  right  place  to  sustain 
combat  operations.  The  environment  required  DLA  to  engage  the  services  in  demand 
planning  and  streamline  their  efficiency  and  accuracy  in  building  business  practices 
(DLA,  2011). 

2.  DLA  Operations 

DLA  provides  a  full  spectrum  of  logistics,  acquisitions,  and  technical  services  to 
the  Army,  Navy,  Air  Force,  Marine  Corps,  and  other  federal  agencies.  DLA  provides 
nearly  all  consumable  items  to  America’s  military  forces  and  85%  of  the  military’s  spare 
parts  (DLA,  2014).  The  consumables  DLA  provides  are  food,  fuel  and  energy,  clothing, 
medical  supplies,  and  construction  equipment  (DLA,  2014).  Over  time,  DLA  has  also 
increased  its  humanitarian  missions  and  is  one  of  the  first  responders  when  a  crisis  occurs 
(DLA,  2011). 

DLA  is  headquartered  in  Fort  Belvoir,  but  it  also  operates  in  48  states  and  28 
countries  to  support  the  warfighter.  It  employs  25,500  civilian  and  military  employees 
and  ranks  within  the  top  15th  percentile  of  Fortune  500  companies,  taking  into  account 
$39  billion  in  sales  and  revenue  and  the  value  of  the  services  they  provide  (DLA,  2014). 
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Although  DLA  has  several  other  operations,  below  are  the  key  responsibilities  relevant  to 
this  study  (DLA,  2014): 

•  Manages  nine  supply  chains  and  nearly  6  million  items 

•  Manages  25  distribution  centers  worldwide 

•  Supports  roughly  2,400  weapon  systems 

•  Administers  the  storage  and  disposal  of  strategic  and  critical  materials 

•  Processes  on  average  98,475  requisitions  and  over  9,000  contract  actions  a 
day 

A  critical  activity  in  the  DLA  enterprise  that  this  study  directly  pertains  to  is  DLA 
Distribution. 

3.  Distribution 

According  to  DLA’s  2012  Annual  Financial  Report,  DLA  Distribution  falls  under 
the  organization’s  supply  management  business  area.  Supply  management  processes 
make  up  99%  of  assets,  liabilities,  revenues,  and  costs  on  the  financial  statement  (DLA, 
2013). 

DLA  Distribution  is  a  field  activity  for  the  agency  and  was  established  in  October 
1997.  Its  headquarters  is  in  New  Cumberland,  Pennsylvania  (DLA,  2013).  In  addition, 
DLA  operates  25  distribution  centers  in  12  states  and  seven  countries  (DLA,  n.d.-c.). 
DLA  Distribution’s  mission  is  to  leverage  global  distribution  networks  to  enable  logistics 
solutions  (DLA,  n.d.-b).  The  Distribution  vision  is  to  be  the  preferred  source  of  global 
distribution  support  for  the  military  services  and  government  agencies  (DLA,  n.d.-b). 

DLA  Distribution’s  primary  functions  consist  of  receiving,  storing,  and  issuing 
material.  Processes  included  in  these  functions  are  off-loading  cargo,  processing  and 
routing,  inspection,  classification,  warehousing,  packaging  and  transportation  planning 
(DLA,  n.d.-c).  DLA  Distribution  refers  to  these  processes  as  workload.  For  this  study, 
workload  is  only  measured  in  issues  and  receipts  as  defined  below  (DOD,  2014): 
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Receipt:  The  processes  and  the  work  required  to  receive  an  item  from  a  customer 
or  supplier  by  a  DLA  distribution  center  to  include  off-loading,  processing/routing, 
inspection,  classification,  and  stocking.  Receipts  are  measured  in  line  items. 

Issue:  The  processes  and  the  work  required  to  issue  an  item  to  a  customer  by  a 
DLA  distribution  center  to  include  processing/routing,  inspection,  packaging,  and 
transportation  planning.  Issues  are  measured  in  line  items. 

B.  OVERVIEW  OF  FORECASTING 

Forecasting  is  a  critical  element  for  business  operations  and  planning.  Supply 
chain  enterprises  use  forecasts  to  estimate  inventory  levels,  work  schedules  and,  in  the 
case  of  DLA,  to  estimate  workload  and  subsequent  staffing  levels  as  well  as  set  prices  for 
customers.  According  to  an  Oracle  Corporation  white  paper  that  discusses  forecasting, 
however,  “The  main  principle  of  forecasting  is  to  find  the  model  that  will  produce  the 
best  forecasts,  not  the  best  fit  to  the  historical  data.  The  model  that  explains  the  historical 
data  best  may  not  be  the  best  predictive  model”  (Oracle  Corporation,  2006,  p.  1). 

1 .  Elements  of  F orecasting 

The  four  broad  elements  of  forecasting  described  below  serve  as  a  framework  for 
working  with  forecast  models.  Strategic  and  tactical  forecasting  are  tied  to  time. 
Quantitative  and  qualitative  forecasting  are  associated  with  managerial  decision  making 
based  on  forecasting. 

a.  Strategic  and  Tactical  Forecasting 

There  are  two  main  types  of  forecasting:  strategic  and  tactical  (Jacobs  &  Chase, 
2011).  Strategic  forecasting  is  a  medium-  to  long-term  outlook  that  is  used  to  help  set  the 
strategy  of  how  to  meet  an  aggregated  workload  (Jacobs  &  Chase,  2011).  Tactical 
forecasting  is  a  short-term  outlook  used  to  make  day-to-day  decisions  of  how  to  meet 
short-term  workload  (Jacobs  &  Chase,  2011).  In  the  case  of  DLA,  tactical  forecasting 
may  be  used  at  individual  distribution  centers  as  they  attempt  to  manage  daily  workload 
fluctuations  and  process  time  requirements.  Strategic  forecasting  is  used  to  set  prices  for 
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the  services  as  well  as  plan  out  medium-  to  long-term  (at  least  six  months)  staffing 
requirements. 

For  the  purposes  of  this  study,  Jacobs  and  Chase’s  timeframe  definitions  (2011) 
are  used  since  they  are  terms  typically  used  in  business  forecasting.  Short  term  is  defined 
as  less  than  three  months.  Medium  term  is  defined  as  three  months  to  two  years.  Long¬ 
term  forecasting  is  defined  as  greater  than  two  years.  DLA  is  attempting  to  forecast 
medium-  to  long-term  workload  using  aggregated  sales  (the  dollar  value  of  items  sold  by 
DLA  to  customers  directly,  known  as  DLA  Direct  Sales).  Workload  is  related  to  these 
sales  but  is  measured  in  the  number  of  transactions  that  occur  in  the  form  of  issues  and 
receipts.  Some  form  of  work  is  required  to  process  these.  The  workload  forecast  is  what 
helps  manage  staffing  requirements  and  helps  DLA  set  prices  for  the  services  because  it 
must  account  for  the  costs  of  those  workers  in  the  price. 

Other  variables,  however,  may  also  influence  workload.  Seasonal  or  cyclical 
changes  to  customer  orders  may  increase  or  decrease  the  amount  of  issues  and  receipts 
distribution  centers  take  on.  A  decreasing  budget  for  a  service  may  not  necessarily  mean 
a  decrease  in  issue  and  receipt  lines.  Perhaps,  the  service  is  buying  items  in  smaller 
increments,  which  actually  increases  the  number  of  receipts  and  issues.  Consumable 
items  that  are  purchased  regardless  of  budget,  cycle,  or  season  may  lend  themselves  to 
historical  data  being  similar  to  forecasted  data. 

b.  Quantitative  and  Qualitative  Forecasting 

Additionally,  policy  changes  and  management  decisions  may  also  affect 
workload.  These  additional  variables  require  qualitative  (judgment)  input  to  the 
forecasting  methodology. 

Managerial  adjustments  are  usually  made  based  on  information  that  is  not 
available  to  the  statistical  model.  Intuition,  expert  opinion,  and  experience  may  facilitate 
a  fairly  accurate  forecast.  In  a  1994  survey  of  forecasters  at  U.S.  corporations,  91%  either 
always  made  adjustments  or  sometimes  made  adjustments  to  their  mathematical  forecast 
results  (Syntetos,  Boylan,  &  Disney,  2009).  The  inevitable  errors  in  mathematical  models 
can  be  ameliorated  by  decisions  managers  make.  The  model’s  job  is  to  get  as  close  to 
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100%  accurate  as  possible  using  data  available.  The  manager’s  job  is  to  make  decisions 
on  how  to  influence  the  remaining  error. 

Although  qualitative  techniques  are  used  to  fine-tune  forecasts,  quantitative 
techniques  are  used  for  the  bulk  of  forecasting  because  they  use  hard  data  that  stem  from 
business  operations.  At  the  same  time,  these  quantitative  methods  need  not  be 
complicated.  A  study  by  Makridakis  (1982)  highlights  that  simple  forecasts  perform  as 
well  as  if  not  better  than  complicated  ones.  “It  is  not  necessarily  the  case  that  complex 
methods  produce  more  accurate  forecasts  than  simple  methods... the  more  noise  or 
randomness  in  the  data,  the  less  important  it  is  to  use  sophisticated  methods” 
(Orchowsky,  Kirchoff,  Rider,  &  Kem,  1986,  p.  7).  An  example  of  a  simple  technique  is 
exponential  smoothing.  A  2006  summary  by  Gardner  of  all  studies  done  since  1985  (65 
total)  that  included  exponential  smoothing  resulted  in  90%  of  them  reporting  that 
smoothing  methods  offered  more  accurate  forecasts  (Syntetos  et  al.,  2009). 

The  acknowledgement  that  simple  forecasts  are  just  as  good  as  complicated  ones 
is  important  for  choosing  the  right  forecast  model  for  businesses.  First,  they  are  easy  to 
use  and  do  not  take  a  lot  of  time;  the  data  that  are  likely  inputs  to  the  model  are  readily 
available  (e.g.,  sales,  demand,  lines).  Second,  they  do  not  require  a  specially  qualified 
person  or  cost  a  lot  to  run  the  models.  We  can  break  down  common  models  that  are  still 
used  prevalently  in  the  supply  chain  industry  into  two  main  types:  time-series  models  and 
causal  models. 

2.  Forecasting  Techniques 

The  following  is  a  basic  overview  of  common  forecasting  techniques. 

a.  Time-Series  Models 

Several  time-series  models  are  prevalent  in  business  planning.  The  three  that  we 
will  look  at  for  this  study  are  moving  average  (MA),  exponential  smoothing  (ES),  and 
auto-regressive  integrated  moving  average  (ARIMA). 

(1)  Moving  Average 
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The  moving  average  technique  uses  the  mean  of  a  designated  number  of  periods 
to  forecast  the  next  period.  The  equation  that  we  use  is  the  following  (Jacobs  &  Chase, 
2011): 

Ft  =  (At-i  +  At-2  +  At-3  +  . . .  At-n)/n 

where 

Ft  =  the  forecasted  value 
At-i  =  actual  value  in  the  previous  period 
At-n  =  actual  value  in  the  nth  period 
n  =  number  of  periods 

The  moving  average  technique  is  very  simple  to  use.  That  simplicity  may  not 
allow  it  to  accurately  forecast  seasonal  data  or  trends,  however  (Orchowsky  et  al.,  1986). 
A  weighted  moving  average  can  also  be  used  to  indicate  the  importance  of  a  previous 
period  to  the  forecasted  value.  The  equation  used  is  the  following  (Jacobs  &  Chase, 
2011): 

Ft  =  Wl  At-l  +  W2At-2  +  W3At-3  +  . . .  WnAt-n 

where 

w  =  weight  of  actual  value  in  previous  period 
(2)  Exponential  Smoothing 

Exponential  smoothing  is  still  the  most  common  forecasting  technique.  It  uses  a 
constant  value,  called  a  smoothing  constant  (a),  that  represents  the  rate  of  reaction  to  a 
difference  between  forecasted  demand  and  actual  demand  for  a  time  period.  The  premise 
of  exponential  smoothing  is  that  the  most  recent  data  is  more  influential  than  older  data 
(Jacobs  &  Chase,  2011).  The  equation  for  simple  exponential  smoothing  (SES)  used  is 
the  following  (Jacobs  &  Chase,  2011): 

Ft  =  Ft-i  +  a(At-i  -  Ft-i) 


where  the  new  forecast  is  equal  to  the  previous  period’s  forecast  plus  a  portion  of  the 
error.  Advantages  of  SES  are  that  it  is  fairly  accurate  (evidenced  by  its  popularity)  and 
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easy  to  use.  It  still  will  lag  behind  any  trends  present,  however,  and  cannot  forecast 
season  data  (Orchowsky  et  al.,  1986). 

Double  exponential  smoothing  (DES)  uses  two  equations  with  two  smoothing 
constants  to  account  for  trend.  The  equations  used  are  the  following  (Orchowsky  et  al., 
1986): 

S’t  =  axt  +  (l-a)S’t-i 
S”t  =  aS’t  +  (l-a)S”t-i 
at  =  2S’t-S”t 
bt  =  (a/l-a)(S’t  -  S”t) 

Ft+m  =  at  +  btm 

where 


S’t  =  the  single  smoothed  value  for  the  current  time  period 

S”t  =  the  double  smoothed  value  for  the  current  time  period 

Xt  =  the  actual  value  for  the  current  time  period 

at  =  the  estimated  level  at  the  current  time  period 

bt  =  the  estimate  of  the  trend  at  the  current  time  period 

m  =  the  number  of  periods  ahead  to  be  forecast 

Ft+m  =  the  forecast  value  for  “m”  periods  ahead 

(3)  Auto-Regressive  Integrated  Moving  Average 

Although  an  auto-regressive  integrated  moving  average  (ARIMA)  model  is  a 
more  complicated  forecasting  method,  it  accounts  for  several  parameters  together, 
including  ES  and  MA,  as  well  as  decomposition  of  trends  and  seasonality.  Since  the  data 
used  in  this  study  is  non-stationary  (exhibits  a  trend)  and  has  seasonal  characteristics,  an 
ARIMA  model  is  appropriate  to  analyze. 

The  ARIMA  model  used  for  this  study  is  the  following  (Hoff,  1983): 
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ARIMA  (p,d,q)  x  (P,D,Q)  s 


where 


p  =  the  number  of  auto-regressive  (AR)  parameters 
d  =  the  number  of  differencing  used  to  make  the  data  stationary 
q  =  the  number  of  MA  parameters 
P  =  the  number  of  seasonal  AR  parameters 

D  =  the  number  of  differencing  to  make  seasonal  patterns  stationary 
Q  =  the  number  of  seasonal  MA  parameters 
s  =  the  number  of  periods  per  season 

(a)  Autocorrelations  (AC)  and  Partial  Autocorrelations  (VAC) 

In  order  to  determine  what  AR,  MA,  or  ARMA  parameters  will  work  best,  the 
data  series’  autocorrelations  (AC)  are  determined,  indicating  how  a  data  series  is  related 
to  itself  over  time  (Hoff,  1983).  Analyzing  theoretical  patterns  of  ACs  and  PACs  can  help 
in  determining  the  number  of  AR  and  MA  parameters,  both  non-seasonal  and  seasonal, 
but  actual  data  series  will  likely  not  adhere  to  these  exact  patterns,  so  comparing  with 
different  parameters  will  narrow  down  the  best  models.  Software  programs,  such  as  JMP 
used  in  this  study,  can  calculate  ACs  and  PACs  as  well  as  the  confidence  intervals  to 
determine  whether  the  ACs  are  significant. 

(b)  Model  Verification 

Too  many  parameters  can  make  an  ARIMA  model  overly  complicated  with  no 
value  added  to  the  model.  As  with  other  techniques,  verification  of  an  ARIMA  model  is 
needed  to  ensure  an  adequate  yet  not  overly  complicated  model.  The  ACs  of  residuals 
can  be  analyzed  to  determine  whether  too  many  or  too  few  AR  and  MA  parameters  are 
used.  The  coefficient  of  determination,  R2,  can  also  be  used,  measuring  how  much  the 
model  accounts  for  variation  in  the  data  series  (Hoff,  1983).  The  R2  value  is  a  quick 
metric  to  compare  models;  the  perfect  fit  for  a  time  series  data  set,  however,  may  not  be 

good  at  predicting  future  values.  It  may  only  be  good  at  fitting  the  original  data  series; 
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therefore,  a  high  R2  value  should  be  combined  with  other  validation  techniques,  which 
are  discussed  in  the  Forecasting  Errors  section. 

The  ARIMA  model  seems  complicated  but,  with  the  assistance  of  statistical 
software,  the  complexity  may  lead  to  a  superior  forecast,  especially  with  trending, 
seasonal  data  that  is  related  to  various  prior  time  periods.  A  critical  element  to  using 
ARIMA  modeling  is  to  create  a  model  that  forecasts  just  as  well  as  it  models  the  original 
data.  Too  many  parameters  leads  to  a  perfect  fit  of  the  original  time  series  but  are  a  poor 
predictor  of  future  values. 

b.  Causal  Models/Regression 

Causal  regression  models  use  independent  variables  other  than  time  to  predict 
dependent  variables.  For  example,  in  the  case  of  DLA’s  current  workload  forecast  model, 
they  use  sales  data  as  an  independent  variable,  or  indicator,  that  causes  a  change  in  the 
dependent  variable,  workload.  The  equation  used  for  linear  regression  is  the  following 
(Jacobs  &  Chase,  2011): 

y  =  Bo  +  Bix 

where 


y  =  the  dependent  variable  being  forecasted 
Bo  =  the  y-intercept 
Bi  =  the  slope 

x  =  the  independent  variable 

The  advantage  of  regression  analysis  is  that  it  takes  into  account  other  factors  that 

may  influence  the  value  being  forecasted,  and  which  do  not  rely  on  trends  over  time.  This 

may  be  a  reason  why  the  current  DLA  workload  model  is  causal -based.  It  is  also  easy  to 

calculate  using  Microsoft  Excel.  The  disadvantage  of  causal  regression  is  that  the 

independent  variable(s)  needs  to  be  identified  and  be  a  leading  indicator(s)  of  the  value 

being  forecasted.  Many  times,  these  other  factors  are  forecasted  themselves,  meaning  the 

variable  being  forecasted  is  dependent  on  another  forecasted  value. 
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3.  Forecasting  Error 

Another  key  element  of  forecasting  is  recognizing  error.  No  forecasting  method  is 
100%  accurate.  Therefore,  the  forecasting  error  is  the  difference  between  what  was 
forecasted  and  what  actually  happened  (Jacobs  &  Chase,  201 1).  In  this  study,  we  look  at 
R2  (as  discussed  in  the  previous  section)  to  determine  how  well  a  model  fits  the  actual 
data,  mean  absolute  deviation  (MAD),  and  mean  absolute  percentage  error  (MAPE). 

MAD  measures  how  far  values  are  from  an  expected  value.  It  is  the  average  error 
in  absolute  terms  (Jacobs  &  Chase,  2011).  The  equation  used  to  calculate  MAD  is  the 
following  (Jacobs  &  Chase,  2011): 

MAD  =  (I|At  -  Ft|)/n 


t  =  time  period  number 
At  =  actual  value  for  the  period  t 
Ft  =  forecast  value  for  period  t 
n  =  total  number  of  periods 

MAPE  relates  the  error  back  to  the  average  value,  which  is  useful  in  determining 
what  percent  error  to  expect  (Jacobs  &  Chase,  2011).  The  following  equation  is  used  to 
calculate  MAPE  (Jacobs  &  Chase,  2011): 

MAPE  =  MAD/ Average  Value 

C.  DLA  FORECASTING 

By  the  nature  of  its  business,  DLA  was  using  simple  forecasting  techniques,  along 
with  industry,  over  fifty  years  ago.  It  began  critiquing  its  forecasting  in  1963.  This 
original  study  recommended  ES  over  a  MA  technique  (Orchowsky  et  al.,  1986).  Several 
years  later,  another  study  was  done  showing  that  DES  was  more  accurate  (Orchowsky  et 
ah,  1986).  Over  the  next  20-30  years,  studies  identified  ES  with  a  smoothing  constant  (a) 
of  .2  to  offer  the  best  demand  forecast,  which  resulted  in  DLA  using  a  modified  and 
slightly  incorrect  version  of  the  DES  model  described  in  the  previous  section  (Orchowsky 
et  ah,  1986). 
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Forecasting  efforts  were  also  conducted  within  the  military  services  that,  until  the 
past  two  decades,  accounted  for  repairable  and  consumable  items.  This  led  the  services  to 
use  program  factors  in  their  forecasting.  In  1983,  Boeing  conducted  a  study  that 
compared  forecasting  techniques  for  Army  and  Navy  data.  The  study  resulted  in  an  eight- 
quarter  MA  technique  to  perform  best  out  of  the  simple  methods.  An  ARIMA  model  also 
performed  well,  but  ES  and  regression  did  not  perform  well  (Orchowsky  et  al.,  1986). 

This  led  to  a  study  done  in  1989  that  looked  at  forecasting  contracting  workload  at 
DLA.  A  causative  model  using  service  activity  consisting  of  equipment  usage  (e.g., 
flying  hours),  personnel,  and  budgetary  activities  (procurement  and  O&M  dollars)  was 
tested.  It  did  not  perform  adequately  enough  to  change  the  current  methods  they  were 
using  (Schwarz  &  Brooks,  1989).  Another  study  in  1991  analyzed  the  impact  of 
decreasing  DOD  budgets  and  consumable  item  transfers  (CIT)  (the  process  was  just 
starting  to  transfer  these  items  from  the  services  to  DLA  control)  on  DLA  “demand 
workload”  (Baker,  1991).  The  recommendation  was  to  use  the  procurement  budget  as  a 
leading  indicator  as  well  as  CIT  in  regression  analysis  to  forecast  demand  in  dollars 
(Baker,  1991). 

By  1996,  consolidation  was  not  complete  yet,  so  the  services  were  still  predicting 
workload  consisting  of  issues  and  receipts  that  were  predicted  to  be  at  DLA  distribution 
depots.  DLA  had  no  formal  model  to  estimate  this  workload.  They  would  predict  based 
off  one  or  two  quarters  and  expand  that  for  the  entire  year.  It  assumed  the  percent  change 
from  previous  year  to  current  year  workload  could  be  applied  to  future  years.  It  then  took 
the  services’  forecasts  and  averaged  it  with  theirs  to  get  the  forecast  for  the  year 
(Warbrick,  1996).  Several  things  made  this  forecasting  method  inadequate.  It  was  based 
on  service  data  that  was  also  forecasted  with  assumptions  of  its  own,  one  of  which  was 
the  assumption  that  the  percent  change  in  sales  was  equivalent  to  the  percent  change  in 
total  requisitions  (issues  and  receipts).  Judgmental  forecasting  was  also  heavily  used  by 
the  services  (Warbrick,  1996).  Warbrick  concluded  in  his  thesis  that  causal-based  factors, 
in  particular,  operations  and  maintenance  (O&M)  budget  and  a  measure  of  operational 
tempo  (OPTEMPO)  could  predict  workload  for  Navy  workload  at  DLA  distribution 
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depots  (Warbrick,  1996).  This  recommendation  influenced  the  linear  regression 
technique  used  in  this  study. 

1.  DLA’s  Current  Workload  Forecasting  Model 

The  following  is  our  understanding  of  the  current  model  used  by  DLA  to  predict 
distribution  workload. 

a.  Introduction 

The  current  model  is  a  causal  model  using  DLA  Direct  Sales  (DD  Sales)  as  the 
independent  variable  and  issues  and  receipts  as  the  dependent  variable  in  a  linear 
regression  analysis.  DD  Sales  result  from  items  that  are  stored  at  a  distribution  facility 
and  require  DLA  direct  labor  to  either  issue  to  a  customer  or  receive  from  a  supplier  to 
replenish  stock.  The  workload  defined  is  the  total  number  of  issues  and  receipts  that 
require  personnel  and  the  subsequent  labor  to  support.  DD  Sales  differ  from  customer 
direct  (CD)  Sales,  which  do  not  require  direct  labor  during  the  transactions.  The  current 
state  of  this  model  is  under  evaluation  as  DOD  and  DLA  environmental  factors,  such  as 
contracted  or  commercial  logistics  support,  may  be  affecting  DLA  Distribution  workload. 
The  current  model  may  no  longer  meet  DLA’s  challenging  environmental  needs  since 
qualitative  techniques  are  used  to  adjust  forecasted  results  that  are  believed  to  be 
inaccurate. 

Some  possible  reasons  why  the  current  model  is  inaccurate  are  the  following: 

•  Sales  are  not  a  good  leading  indicator  of  lines  received  or  issued.  For 
example,  if  service  budgets  are  decreasing,  it  is  possible  that  services  are 
ordering  smaller  quantities  more  frequently,  which  would  actually  offer  no 
change  or  an  increase  to  the  issue  and  receipt  workload  involved. 

•  Sales  are  being  estimated  themselves,  leading  to  compounding  of  forecast 
errors  (Lee,  Padmanabhan  &  Whang,  1997).  Sales  are  taken  from  the 
services  and  different  supply  chains  based  on  any  number  of  factors  that 
the  services  deem  will  influence  their  purchase  figures. 

•  The  decrease  in  DOD  budgets  is  having  a  significant  effect  on  DLA’s 
ability  to  forecast  workload. 
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•  The  model  aggregates  sales,  issues  and  receipts.  Maybe  one  model  is  not 
the  best  fit  for  all  the  supply  chain  sales.  Forecast  error  may  be  driven  by 
structural  change  in  one  supply  chain,  while  the  others  remain  more 
predictable. 

•  Other  external  factors  may  influence  the  workload  data  that  are  not 
correlated  with  sales.  There  may  be  a  seasonality  characteristic  or  a 
relationship  with  other  external  indicators. 

DLA’s  causal-based  regression  model  attempts  to  predict  the  annual  distribution 
workload  measured  in  terms  of  issues  and  receipts  from  sales.  The  workload  forecast 
projections  for  each  fiscal  year  (FY)  are  based  on  the  previous  twelve  months  of  actual 
DLA  Direct  (DD)  sales,  processing  and  storage  workload  (DOD,  2014,  pp.  81-82).  Each 
supply  chain  provides  sales  estimates  and  percentage  of  DD  sales  stored  at  distribution 
centers.  Only  the  sales  from  four  supply  chains  are  used  to  determine  workload  across 
DLA:  Maritime,  Aviation,  Land,  and  Industrial  Hardware.  The  other  supply  chains  are 
reasoned  to  be  inconsequential  to  predicting  workload.  The  estimates  of  the  four  primary 
supply  chains  are  totaled  to  provide  yearly  sales  projections.  The  sales  estimates  are 
applied  to  the  regression  analysis,  predicting  future  fiscal  year  estimates  for  total  number 
of  lines.  DLA  uses  the  forecasted  workload  values  to  make  decisions  on  staffing 
requirements,  distribution  center  spending  plans,  and  proper  rate  setting  to  recover  costs. 

b.  Current  Model  Assumptions 

The  following  assumptions  are  included  in  the  current  regression  model: 

1 .  The  correlation  between  sales  and  workload  is  strong  enough  to  accurately 
predict  workload. 

2.  The  estimated  sales  projections  provided  by  each  supply  chain  are 
sufficiently  accurate. 

3.  Maritime,  Aviation,  Land,  and  Industrial  Hardware  supply  chains  have  the 
only  significant  impact  on  workload,  and  other  supply  chains  (Clothing  & 
Textiles,  Medical,  Subsistence,  and  Construction  &  Equipment)  do  not 
have  a  significant  impact  on  workload. 

4.  Sales  numbers  are  adjusted  into  comparable  constant  dollar  amounts. 
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c.  Causal  Indicators 

The  current  regression  analysis  is  dependent  on  the  information  provided  by  each 
supply  chain  and  the  four  military  services.  The  supply  chains  provide  sales  forecasts 
based  on  system-generated  modeling,  customer  input,  and  strategic  policy  decisions.  The 
customer  inputs  are  provided  by  each  service  and  are  projected  demand  requirements 
based  on  operational  planning  estimates.  The  sales  estimates  included  in  the  workload 
regression  model  are  further  segregated  to  the  percentage  of  DD  sales  stored  at 
distribution  centers.  The  sales  estimates  are  adjusted  by  the  percentage  of  historic  sales 
derived  from  inventory  currently  in  storage  at  distribution  centers.  The  estimates  are  then 
entered  into  the  regression  model,  which  generates  future  total  FY  distribution  issues  and 
receipts. 

DLA  charges  its  customers  based  on  the  issue  or  receipt  of  material  from  its 
distribution  centers.  The  issues  and  receipts  are  converted  into  a  Line  Charge  according 
to  the  distribution  net  landed  cost  (NLC)  method  for  setting  customer  rates.  The  ultimate 
goal  is  to  fully  recover  all  cost  associated  with  its  distribution  operations.  DLA  attempts 
to  balance  cost  recovery  with  providing  a  fair  and  equitable  price  to  each  individual 
customer  (DLA,  n.d.-a).  The  set  rate,  or  price  to  customers,  directly  affects  the  current 
year’s  sales  totals.  The  current  rate,  in  turn,  affects  the  future  estimated  sales  data,  which 
is  the  causal  indicator  for  total  lines.  The  process  is  outlined  in  a  flowchart  in  Figure  1. 
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Figure  1 .  The  process  flow  in  predicting  DLA  Distribution  workload 


d.  Program  Budget  Review  16  Forecasting  Example 

The  Program  Budget  Review  16  (PBR16)  forecasting  regression  analysis  was 
conducted  in  March  2014.  DLA  used  actual  data  from  the  previous  12  months,  March 
2013  through  February  2014,  to  create  a  model  predicting  future  workload  (DLA,  n.d.-d). 
This  is  shown  in  Table  1. 


Month  /  FY 

DLA  Direct  (DD) 
Sales  @  Cost 

Total  Receipts  & 
Issues 

Oct  -  FY  1 4 

318,324 

1,078,625 

Nov  -  FY  14 

290,521 

1,361,808 

Dec  -  FY  14 

281,142 

744,818 

Jan  -  FY  14 

322,367 

968,600 

Feb  -  FY  14 

322,495 

1,028,436 

Mar  -  FY  13 

434,086 

1,327,576 

Apr  -  FY  13 

241,718 

1,408,681 

May  -  FY  13 

316,086 

1,390,207 

Jun  -  FY  13 

336,056 

1,222,920 

Jul  -  FY  13 

305,843 

1,147,700 

Aug  -  FY  13 

330,255 

1,329,556 

Sep  -  FY  13 

347,319 

1,273,738 

Total 

3,846,212 

14,282,665 

Table  1 .  DD  Sales  and  I&R  data  provided  in  the  PBR16  regression  model 

(from  DLA,  n.d.-d) 
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The  previous  twelve  months  of  data  was  analyzed  to  produce  a  simple  linear 
equation.  The  linear  equation  becomes  the  model  to  predict  future  distribution  lines.  The 
equation’s  independent  variable,  denoted  by  x,  is  the  total  monthly  sales  estimate  for  each 
FY.  The  equation’s  dependent  variable,  denoted  by  y,  is  the  predicted  number  of  total 
monthly  distribution  lines  for  each  FY.  Again,  these  are  based  on  the  four  supply  chains 
having  the  largest  impact  on  workload.  The  equation  produced  from  the  PBR  16  analysis 
is  below: 

y  =  0.618  x  +  992081.437 

See  Figure  2  below  for  the  graphical  output  of  the  linear  regression. 


Figure  2.  Monthly  DD  Sales  for  the  four  primary  supply  chains  (Avn,  Land, 
Maritime,  Ind  HW)  plotted  to  generate  a  linear  equation  used  to  predict 
future  distribution  workload  (Monthly  I&R)  (from  DLA,  n.d.-d) 


The  future  total  FY  estimated  sales  are  entered  into  the  equation  to  produce  the 
annual  estimated  workload  predictions.  The  process  is  conducted  every  year  and  sales  are 
in  current  FY  dollar  amounts,  FY14  for  PBR  16  projections  (Table  2). 
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Fiscal 

Year 

Esimate  DD  Sales 
(Constant  $) 

Estimated  Workload 
(Reciepts  &  Issues) 

FY  14 

3,995,428 

14,374,147 

FY  15 

3,857,904 

14,289,157 

FY  16 

3,746,471 

14,220,291 

FY  17 

3,696,077 

14,189,147 

FY  18 

3,671,969 

14,174,249 

FY  19 

3,651,238 

14,161,437 

FY  20 

3,624,434 

14,144,872 

Table  2.  PBR16  workload  projections  (from  DLA,  n.d.-d) 


e.  Accuracy 

The  accuracy  for  the  current  model  can  be  evaluated  by  looking  at  the  R2  to 
determine  the  model  forecasting  precision,  the  MAD  and  MAPE  to  determine  the  error  in 
the  models  prediction.  The  data  analysis  generated  an  R2  of  0.019.  The  R2  is  a  measure  of 
the  relationship  between  the  dependent,  x  variable,  and  independent,  y  variable.  An  R2 
close  to  “1”  indicates  a  strong  linear  relationship  between  the  two  variables.  An  R2  close 
to  “0”  indicates  a  weak  relationship.  DLA’s  current  R2  is  almost  zero.  Sales,  as  currently 
measured,  seem  to  explain  almost  none  of  the  variation  in  workload.  The  MAD  is 
158,421  lines  per  month  and  the  MAPE  is  13.31%  error  between  actual  and  predicted 
lines  per  month.  These  will  be  used  later  to  compare  the  current  model  to  different 
forecasting  techniques. 

2.  How  the  Model  Is  Used 

DLA  sets  its  rates  and  staffing  levels  based  on  the  total  number  of  lines  estimated 
for  a  FY.  An  individual  issue  or  receipt  can  be  referred  to  as  a  line.  DLA  Distribution 
uses  the  predicted  workload  to  estimate  full  time  equivalent  (FTE)  employee 
authorizations.  Staff  estimates  are  based  on  a  productivity  goal  defined  as  Lines  (Issue  or 
Receipt)  per  Paid  Equivalent  or  LI/PE.  The  LI/PE  is  determined  by  using  the  historical 
average  productivity  achieved  over  past  FYs.  DLA  Distribution  takes  the  forecasted 
average  monthly  workload,  divided  by  a  calculated  productivity  goal  to  determine  FTE 
employees  authorized.  The  resulting  estimated  FTE  staffing  is  included  in  the  cost 
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calculation  and  budgeting  projections  for  DLA  Distribution  as  a  whole.  DLA 
Distribution’s  PBR  forecast  and  rates  are  set  two  years  out. 

The  PBR  projections  are  further  apportioned  to  each  site  based  on  the  site’s 
historical  percentage  of  total  overall  projected  workload.  DLA  Distribution  does  not 
allocate  the  actual  workload  to  individual  distribution  centers.  DLA  HQ  and  individual 
services  determine  the  workload  allocation  based  on  material  stocks  at  each  distribution 
center. 

DLA  Distribution  HQ  uses  feedback  provided  by  each  distribution  center  to 
finalize  site-specific  workload  projections  in  the  year  of  execution.  The  workload 
projections  are  then  used  to  calculate  staffing  requirements  and  budget  allocations.  The 
staffing  targets  and  budget  allocations  are  distributed  to  each  distribution  center. 
Individual  distribution  center  allocations  are  made  in  the  year  of  execution.  The 
allocations  include  a  spending  plan  (budget)  and  staffing  authorizations  (FTE).  If  the 
workload  percentage  allocations  change  from  the  previous  year,  then  the  adjustments  to 
both  budget  and  staffing  will  be  made  to  match  the  percentage  change. 

D.  SUMMARY 

This  section  provided  a  definition,  key  elements,  and  history  of  forecasting  in 
industry.  By  looking  at  historical  studies  summarizing  forecasting  techniques,  we  present 
models  that  are  likely  to  provide  DLA  Distribution  with  an  updated,  more  accurate,  and 
easy  to  use  forecasting  technique  as  well  as  key  advantages  and  disadvantages  of  each. 
We  also  described  the  chronology  of  forecasting  at  DLA,  up  until  the  current  workload 
forecasting  model.  Lastly  we  described  the  current  model  and  explored  the  possible 
reasons  why  it  may  not  accurately  predict  workload,  which  we  hope  to  remedy  through 
this  project. 


21 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


22 


III.  METHODOLOGY 


The  methodology  describes  our  process  from  the  recognized  problem  with  the 
current  DLA  Distribution  workload  forecast  model  through  analyzing  available  data  to 
selecting  a  suitable  forecast  model  and  predicting  DLA  Distribution  future  workload. 
This  process  has  four  main  tasks:  1)  collection  and  analysis  of  available  relevant  data,  2) 
determination  of  forecasting  techniques,  3)  analysis  and  comparison  of  the  models,  4) 
sensitivity  and  simulation  analysis. 

A.  FOUR-STEP  PROCESS 

1.  Collection  and  Analysis  of  Available  Relevant  Data 

Our  study  uses  historical  monthly  workload  figures  (issues  and  receipts)  over  the 
past  ten  fiscal  years  (2004-2013).  This  gives  us  a  suitable  number  of  data  points  to 
analyze,  with  120  data  points  in  the  series.  DLA  Operations  Research  and  Resource 
Analysis  (DORRA)  provided  us  with  ten  years  of  issues  and  receipts  broken  down  by 
month  and  by  supply  chain  and  service. 

In  addition  to  the  data  DORRA  gave  us,  we  recorded  the  O&M  annual  budgets 
for  all  ten  years  for  analysis  as  a  leading  indicator  of  workload.  The  O&M  budgets  are 
what  the  services  use  to  spend  money,  and  buying  supplies  and  parts  from  DLA  is  part  of 
those  expenditures.  To  narrow  down  the  potential  relationship,  we  only  looked  at  four 
O&M  Budget  Activities:  Operation  and  Maintenance,  Operating  Forces,  Mobilization, 
and  Operation  Support.  Purchases  from  DLA  would  generally  originate  from  these  funds. 

To  better  understand  the  data,  we  graph  the  data  points  chronologically  from 
October  FY04  (October  2003  on  our  graphs)  to  September  FY13  (September  2013  on  our 
graphs)  in  Excel  and  analyze  for  trend  patterns  and  seasonality.  The  software  tool,  JMP, 
also  gives  us  good  indicators  of  trend  and  seasonality  through  analysis  of  the 
autocorrelations.  Once  our  understanding  of  the  data  series  is  sufficient,  we  decide  on 
methods  to  forecast  workload  for  the  two-year  PBR  cycle  that  DLA  sets. 
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2.  Determination  of  Forecasting  Techniques 

Based  on  the  workload  data  we  have  as  well  as  results  from  previous  studies  on 
forecasting  techniques,  we  are  using  an  ARIMA  model  as  our  primary  technique  but 
comparing  this  with  three  other  techniques:  double  exponential  smoothing  (DES), 
moving  average  (MA)  and  linear  regression.  We  are  using  an  ARIMA  model  because  it 
encompasses  the  concept  of  exponential  smoothing  with  the  autoregressive  and  moving 
average  terms  as  well  as  accounting  for  trend  (stationarity)  and  seasonality.  Since  our 
data  series  exhibits  trend  and  seasonality  an  ARIMA  model,  although  complicated,  has 
the  capacity  to  offer  a  refined,  more  detailed  forecast.  Since  it  is  complicated,  three 
ARIMA  models  are  compared.  The  most  commonly  used  seasonal  ARIMA  model, 
(0,l,l)x(0,l,l)  (Nau,  2014)  is  compared  against  two  ARIMA  models  that  we  determine 
to  best  fit  the  data  series  and  predict  future  values.  Our  best-fit  determination  is  based  on 
analysis  of  autocorrelations  and  selection  of  seasonal  and  non-seasonal  AR  and  MA 
parameters  facilitated  by  JMP’s  ARIMA  Model  Group  tool  that  determines  a  best-fit 
model  for  the  data  series. 

We  compare  our  ARIMA  models  with  a  DES  technique  because  DES  is  shown  to 
be  a  good  model.  Gardner’s  2006  study,  as  noted  earlier,  illustrates  the  effectiveness  of  a 
simple  exponential  smoothing  technique  in  forecasting.  In  Orchowsky  et  al.’s 
summarization  of  previous  study  results,  DES  is  shown  to  be  a  good  technique  used  by 
DLA  in  the  past.  To  compare  complexity  and  simplicity  further,  we  are  also  using  a 
moving  average  technique.  Moving  average  is  one  of  the  simplest  forecasting  methods, 
and  was  shown  by  the  Boeing  study  to  forecast  Army  and  Navy  data  well. 

Lastly,  we  are  using  a  linear  regression  model  with  O&M  dollars  as  the 
independent  variable  to  deduce  whether  another  leading  indicator  may  replace  and  be 
better  than  sales.  The  assumption  in  this  approach  is  that  the  services’  O&M  budget  is  a 
good  representation  of  DLA’s  customer  buying  power. 

To  validate  that  our  ARIMA  model  is  a  good  fit  for  workload  data  and  the 
complexity  of  it  does  actually  improve  the  forecast,  comparing  it  against  these  three  other 
forecasting  techniques  is  warranted. 
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3.  Analysis  and  Comparison  of  the  Models 

We  use  JMP  and  Excel  to  help  our  analysis  of  each  technique.  The  JMP  tool  can 
determine  the  coefficients  for  ARIMA,  DES,  and  moving  average  parameters.  The  linear 
regression  can  just  as  easily  be  done  in  Excel.  To  determine  this  best  model,  comparison 
criteria  are  needed. 

To  compare  the  accuracy  of  the  fitted  ARIMA  model  with  the  other  five,  we 
measure  forecast  error  by  calculating  MAD  and  MAPE.  To  compare  precision  and  how 
well  each  model  explains  variation,  we  measure  the  coefficient  of  determination,  R2. 
Table  3  is  completed  in  the  Results  section: 


Model 

MAD 

MAPE 

R2 

ARIMA  (2,1,2)x(2,0,1) 

ARIMA  (l,l,l)x(l, 0,1) 

ARIMA  (0,l,l)x(0,l,l) 

DES 

MA 

O&M  Regression 

Current  Regression 

Table  3.  Model  comparison  matrix 


From  the  results  of  our  comparison,  we  select  the  best  performing  models  and 
conduct  a  sensitivity  analysis. 

4.  Sensitivity  and  Simulation  Analysis 

We  select  best-fit  models  based  on  the  above  criteria  and  analyze  how  well  they 
forecast  under  various  situations.  We  use  back-casting  and  upward  and  downward  spikes 
to  determine  how  well  our  selected  models  perform.  Since  DLA  Distribution  uses  a  two- 
year  cycle  (PBR  cycle)  to  set  rates  and  staffing  targets,  we  back-cast  two  years. 

We  analyze  an  incremental  approach  with  back-casting.  Starting  with  the  first 
three  years  of  data,  called  the  training  set  (Hyndman,  2010),  we  incrementally  include 
another  year  for  each  iteration  as  we  back-cast  for  the  following  two  years,  called  the  test 
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set  (Hyndman,  2010).  For  example,  in  FY07  we  use  training  set  data  from  FY04-FY06 
and  predict  FY08  and  FY09.  The  next  iteration  includes  four  years  of  data  and  predicts 
FY09  and  FY10.  This  is  done  until  we  reach  the  last  two  years  of  data,  FY12  and  FY13. 

We  look  at  our  accuracy  and  precision  criteria  again  and  determine  the  confidence 
level  of  our  models  in  predicting  workload.  When  comparing  the  forecasted  values  with 
actual  values  in  the  test  set,  we  use  average  error  and  percent  error  of  aggregated  yearly 
workload  instead  of  MAD  and  MAPE,  since  DLA  Distribution  uses  an  aggregated  yearly 
workload  value  to  plan  with.  Based  on  forecast  result  accuracy,  through  average  error  and 
percent  error  comparison,  we  select  the  best  three  models  for  further  analysis. 

To  simulate  future  changes  and  determine  how  our  selected  models  react  to  a 
future  that  does  not  look  like  the  past,  we  develop  two  scenarios:  an  uptick  in  issues  and 
receipts  over  the  next  four  years  and  a  downtick  utilizing  the  existing  down-trending  data. 
We  use  the  absolute  value  of  the  four  largest  annual  percent  changes  in  workload  over  the 
data  series  as  a  basis  for  a  simulated  uptick  and  allocate  the  simulated  data  across  each 
month  based  on  historical  monthly  allocation  percentage.  We  examine  the  down  trending 
data  from  FY10  through  FY13.  Analysis  of  each  scenario  will  determine  whether  our 
models  are  still  performing  well.  Again,  we  compare  these  final  three  models  by  average 
error  and  percent  error  over  the  two-year  PBR  cycle  portion  of  the  test  set,  and  select  the 
best  model. 

Finally,  we  compare  our  forecast  numbers  from  our  selected  model  to  the  current 
model’s  forecasted  numbers  and  see  what  would  have  been  reported  if  our  model  was 
used.  To  do  this,  we  replace  the  actual  workload  values  with  the  actual  workload  values 
we  used  from  DORRA  and  re-run  the  current  regression  model  to  determine  the  predicted 
workload.  This  allows  us  a  comparison  between  what  the  current  model  predicted  and 
what  our  proposed  model  predicts. 

B.  PRESENTATION  OF  RESULTS 

The  result  is  a  range  of  workload  values  that  our  model  predicts  within  the 
confidence  limits.  The  risks  associated  with  values  on  the  periphery  of  the  confidence 
limits  relate  to  staffing  levels,  spending  plans,  and  rate-setting.  We  use  the  selected  model 
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and  predict  workload  for  the  next  two  years  and  use  those  figures  to  walk  through  the 
process  of  determining  FTE  requirements  and  potential  savings  that  may  occur. 

We  present  our  findings  to  DLA  Distribution  and  explain  the  risk  factors 
associated  with  this  forecast  and  any  forecast.  Their  decisions  need  to  account  for  the  risk 
of  a  model  not  being  100%  accurate.  Ultimately,  the  goal  is  to  provide  DLA  with  a 
practical  model. 

C.  SUMMARY 

This  chapter  described  the  relevant  data  we  used  in  building  a  workload 
forecasting  model  for  DLA  Distribution  and  how  we  determined  the  four  forecasting 
techniques  used.  A  comparison  of  the  techniques  is  described  using  the  criteria  of  MAD, 
MAPE,  and  R2  to  determine  the  best  model.  Lastly,  we  described  testing  how  well  our 
forecast  model  performs  by  using  back-casting  and  future  workload  scenarios.  The  next 
chapter  will  present  the  results  of  this  methodology. 
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IV.  ANALYSIS  AND  RESULTS 


This  chapter  describes  our  analysis  of  the  data  series  and  results  of  implementing 
the  described  methodology.  Through  test  and  comparison  of  forecasting  models,  we 
reach  a  final  forecast  model  to  compare  with  the  current  DLA  regression  model  the 
workload  values  that  would  have  been  forecast  if  our  model  was  used. 

A.  DATA  ANALYSIS 

1.  Describing  the  Data 

The  monthly  workload  data  used  to  conduct  our  analysis  was  gathered  from  an 
analyst  at  DORRA.  The  data  has  service  workload  broken  down  among  Navy,  Army,  Air 
Force,  Marines,  DLA,  and  Other.  Within  each  service,  data  was  further  broken  down 
among  the  supply  chains,  where  the  workload  data  was  identified  as  a  receipt  or  issue. 
We  conducted  analysis  using  consolidated  monthly  workload,  which  accounted  for 
receipts  and  issues  amongst  all  supply  chains  and  services.  The  data  consisted  of  131  data 
points  and  13  complete  fiscal  years  from  October  2003  to  August  2014.  Analysis 
comparing  fiscal  years  does  not  include  FY14,  because  we  did  not  have  September  2014 
data.  When  appropriate,  however,  we  utilized  all  the  data  points  available.  From  FY2000 
to  2003  we  were  only  able  to  account  for  total  yearly  workload  numbers,  because 
monthly  data  was  not  available.  These  data  points  were  not  used  in  our  forecast 
modeling,  but  the  yearly  values  provided  us  with  supporting  evidence  used  to  conduct  our 
sensitivity  analysis. 

DORRA’ s  issues  and  receipts  used  in  our  analysis  are  slightly  larger  than  the 
receipts  and  issues  used  by  DLA  in  the  existing  model,  as  seen  in  Table  4.  DORRA’s 
numbers  consider  gross  issues  and  receipts,  while  DLA’s  numbers  are  adjusted  for 
specific  coded  lines  that  are  not  part  of  DLA  Distribution’s  net  landed  cost  model.  Since 
we  consistently  use  DORRA  numbers  throughout  the  analysis,  our  models  are  suitable 
for  forecasting  workload  over  PBR  cycles. 
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j  issue  and  Receipt  Comparison 

Year 

DORRA 

DLA 

DORRA's  percentage 
difference 

2010 

22,706,614 

21,636,406 

+4.9% 

!  2011 

21,204,852 

20,871,565 

+1.6% 

'  2012 

19,429,227 

18,433,148 

+5.4% 

2013 

17,013,701 

15,801,816 

+7.7% 

Table  4.  DORRA  vs  DLA  issue  and  receipt  number  differences 


The  graphs  in  Figures  3  through  8  represent  DLA  Distribution’s  workload  (WL) 
profile  based  on  proportion  of  total  and  service  specific  issues  and  receipts.  Our  issues  and 
receipts  requisitions  are  counted  based  on  the  customer  and  not  classified  by  the  service 
owner.  For  example,  stock  owned  by  DLA  but  requisitioned  by  the  Army  will  be  counted 
as  an  Army  issue,  not  a  DLA  receipt.  Figure  3  shows  the  issues  and  receipts  by  service. 


Service  WL  Spread 
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Figure  3 .  Issues  and  receipts  by  service 


Figure  4  depicts  the  allocation  of  workload  as  a  percentage  to  each  service  over  an 
1 1-year  period.  Amy’s  workload  matches  best  with  the  trend  of  DLA’s  total  workload  and 
represents  the  decline  in  total  workload  where  deployments  and  budgets  shrunk.  The  Navy 
and  Air  Force  workload  as  a  percentage,  however,  remained  relatively  constant  over  time. 
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Issue  /  Receipt  Alfocation 
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Figure  4.  Issue  and  receipt  allocation  200A-2014 


Figure  5  displays  the  yearly  supply  chain  workload  from  fiscal  year  2004  to  fiscal 
year  2014.  Industrial  Hardware  (IH)  is  a  newly  recognized  supply  chain  that  did  not 
create  additional  DLA  line  items.  The  line  items  within  the  newly  formed  supply  chains 
were  reclassified  from  existing  supply  chains.  Except  for  IH,  the  aviation  supply  chain 
was  the  only  supply  chain  to  show  a  significant  change.  Aviation’s  workload  reduced  at  a 
much  larger  rate  than  all  other  supply  chains  from  201 1  to  2014.  Aviation  went  from  3 1% 
to  23%  of  DLA’s  total  workload  in  three  years.  This  significant  change  could  be  related 
to  the  reduction  of  overseas  contingency  operations  or  due  more  in  pail  to  the  items  being 
reclassified  into  IH. 
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Service  Workload  Allocation 
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Figure  5.  Total  workload  allocation  by  Service  percentage 

Figure  6  displays  the  total  workload  percentage  for  each  supply  chain  during 
FY14,  excluding  August.  Four  supply  chains  represent  over  70%  of  DLA’s  workload. 
These  four  supply  chains:  Maritime,  Land,  Aviation,  and  IFF  have  a  large  influence  on 
DLA’s  total  workload.  Therefore,  understanding  the  causes  of  workload  within  these 
supply  chains  could  result  in  a  better  understanding  of  DLA  future  total  workload. 
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Yearly  Supply  Chain  Workload  ■  Subsistence  (SUBS) 
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Figure  6.  Total  workload  allocated  by  supply  chain  2004-20 1 4 

Figure  7  displays  the  historic  average  monthly  allocation  provided  from  10  years 
of  data.  The  monthly  standard  deviation  is  calculated  and  presented  in  the  chart  below  the 
graph,  which  displays  a  minimal  monthly  variation  from  year  to  year.  The  linear  trend 
line  represented  by  the  blue  dotted  line  demonstrates  an  upward  trend  of  workload 
throughout  any  given  year.  The  upward  trend  is  a  result  of  the  DOD  quarterly  spending 
process. 

Finally,  Figure  8  shows  the  historic  monthly  workload  allocation  between  2004 
and  2014. 
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FY  14  SUPPLY  CHAIN  ALLOCATION 
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Figure  7.  FY  14  workload  allocation  by  supply  chain 
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Figure  8.  Historic  monthly  workload  allocation  2004—2014 
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B.  MODEL  FIT 

1.  Model  Description 

In  additional  to  describing  the  models,  we  included  the  JMP  software  outputs, 
which  contain  the  model  summary,  parameter  estimates,  and  forecast  graphs  of  our 
models. 


a.  ARIMA  (2, l,2)x(2, 0,1)12 

The  following  describes  the  process  we  used  to  appropriately  fit  the  available  data 
to  each  ARMIA  model. 

(1)  Set  value  for  “d” 

We  first  recognized  a  downward  trend  in  the  data  as  shown  below  by  the  data 
series  and  auto  correlation  plots  (Figures  9  and  10). 
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Figure  9.  Monthly  workload  data  series  graph 
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Figure  10.  Workload  data  series  autocorrelations 


To  make  the  data  stationary,  we  added  a  first  difference  transformation  (d).  The 
subsequent  residual  graph  (Figure  11)  and  differencing  auto  correlation  plots  (Figure  12) 
show  data  stationarity. 
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Figure  1 1 .  Differenced  residual  graph 
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Figure  12.  Differenced  data  autocorrelations 
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(2)  Set  value  for  “D” 

The  auto  correlation  plots  from  the  differenced  data  supports  our  findings  that  our 
data  does  have  seasonality  (Figure  12).  In  particular,  every  12  months  or  periods  there  is 
correlation  between  the  same  months  from  a  previous  year.  Therefore,  we  use  a  12-period 
season  in  our  ARIMA  model.  Additionally,  the  data  series  exhibits  no  trend  to  the 
seasonal  pattern,  which  is  supported  by  the  differenced  data  graph  (Figure  11). 
Consequently,  the  seasonal  differencing  order  (D)  is  zero. 

(3)  Set  AR  and  MA  parameters  (“p,q  and  P,Q”) 

The  autocorrelation  and  partial  auto  correlation  plots  for  the  differenced  data  do 
not  provide  a  clear  signature  of  the  number  of  AR  or  MA  parameters  needed.  There  is  no 
evidence  from  analysis  of  autocorrelations,  however,  to  suggest  more  than  2  AR  or  MA 
seasonal  and  non-seasonal  parameters  (Figure  12).  Additionally,  too  many  parameters 
may  lead  to  a  perfect  fit  of  the  original  time  series,  but  may  be  a  poor  predictor  of  future 
values  (Nau,  2014).  Using  the  ARIMA  Group  model  function  from  JMP,  we  determined 
that  our  best-fit  model,  based  on  an  R2  of  .868,  was  ARIMA  (2,1,2)  x  (2,0,1)12.  Figures 
13  through  15  illustrate  the  R2,  MAD,  and  MAPE  of  this  ARIMA  model  as  well  as 
parameter  value  estimates,  and  show  a  graph  of  the  model’s  outputs,  with  the  red  line 
showing  the  predicted  values  and  the  blue  lines  on  either  side  showing  the  confidence 
interval. 


Model  Summary 


DF 

Sum  of  Squared  Errors 
Variance  Estimate 
Standard  Deviation 


1.0773e+12  Invertible  Yes 
8830020772 
93968.1902 


122  Stable  Yes 


Akaike's  'A' Information  Criterion  3382.4913 


Schwarz's  Bayesian  Criterion 

RSquare 

RSquare  Adj 

MARE 

MAE 

-2LogLikelihood 


3405.43158 
0.86767398 
0.86008679 
4.21985315 
77  0  34.6644 
3366.4913 


Figure  13.  R2,  MAD  (MAE),  and  MAPE  for  ARIMA  (2,l,2)x(2,0,l)12 
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Parameter  Estimates 


Term 

Factor 

Lag 

Estimate 

Sid  Error 

1  Ratio 

Prob&|t| 

Constant 

AR1.1 

1 

I 

0.434 

0.016 

26.78 

<  0001* 

Estimate 

AR1.2 

1 

2 

0.-1 -17 

0.0041263 

28,41 

<.0001* 

-0.0058436 

AR2.12 

2 

12 

-1.230 

1.6221  e-6  788860 

<.0001* 

AR2.24 

2 

24 

-0.280 

3.4429e-7 

-8e+5 

<.0001* 

MA1.1 

1 

1 

1.-108 

0.030 

36.54 

<.0001* 

MAI  ,2 

1 

2 

-0.234 

0.012 

-20.27 

<.0001* 

MA2.12 

2 

12 

0.998 

0.0005488 

1818.8 

<.0001* 

Intercept 

1 

0 

-8100.666 

9494.890 

-0.85 

0.3952 

Figure  14.  Parameter  estimates  for  p,q,P,Q  for  ARIMA  (2,l,2)x(2,0,l)12 


Figure  15.  Forecast  graph  for  ARIMA(2,l,2)x(2, 0,1)12 


b.  ARIMA  (0,l,l)x(0, 1,1)12 

We  conducted  further  testing  with  more  than  one  ARIMA  model  because  the 
level  of  fitness  depicted  by  R2  is  not  always  a  good  predictor  of  future  data.  We  chose 
ARIMA  (0, 1 , 1  )x(0, 1,1)12  because  it  is  one  of  the  most  common  seasonal  ARIMA 
models  used  (Nau,  2014).  Figures  16  through  18  illustrate  the  R2,  MAD,  and  MAPE  of 
this  ARIMA  model  as  well  as  parameter  value  estimates,  and  show  a  graph  of  the 
model’s  outputs. 
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Model  Summary 

DF  115  Stable  Yes 

Sum  of  Squared  Errors  9.9116e+11  Invertible  No 

Variance  Estimate  861 8766270 

Sta  n  d  a  rd  D  e  vi  ati  on  92  837 .3 1 08 

Akaike’s  'A"  Information  Criterion  3066.50131 


Schwarz's  Bayesian  Criterion 

RSquare 

RSquare  Adj 

MARE 

MAE 

-2  L  o  g  L  i  ke  I  i  hood 


3074.81337 

0.83922819 

0.83643216 

4.31113732 

76859.4029 

3060.50131 


Figure  16.  R2,  MAD  (MAE),  and  MAPE  for  ARIMA  (0,l,l)x(0, 1,1)12 


Parameter  Estimates 

Perm  Factor  Lag  Estimate  Std  Error  t  Ratio  Prob*|t|  Constant 

MA1.1  1  1  0.6567  0.0688  9.54  <  0001*  Estimate 

MA2.12  2  12  1.0000  0.2756  3.63  0.0004*  -184.74476 

Intercept  1  0  -184.7448  931.1324  -0.20  0.8431 


Figure  17.  Parameter  estimates  for  p,q,P,Q  for  ARIMA  (0,l,l)x(0, 1,1)12 


Figure  18.  Forecast  graph  for  ARIMA(0,l,l)x(0,l,l)12 


c.  ARIMA  (l,l,l)x(l, 0,1)12 

The  third  model  chosen  was  ARIMA  (1,1,1  )x(  1 ,0, 1)12  because  it  took  our 
original  model  and  simplified  it  by  removing  AR  and  MA  terms  that  could  be  over- fitting 
the  data,  but  without  losing  seasonality  for  future  values.  Figures  19  through  21  illustrate 
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the  R2,  MAD,  and  MAPE  of  this  ARIMA  model  as  well  as  parameter  value  estimates, 
and  show  a  graph  of  the  model’s  outputs. 


Model  Summary 


DF 

Sum  of  Squared  Errors 
Variance  Estimate 
Standard  Deviation 
Aka  ike's  A.'  Inf  orrnation  Criterion 


125  Stable  Yes 
1.087e+12  Invertible  Yes 
S695658209 
93250.5-132 
3384.67746 


Schwarz's  Bayesian  Criterion 

RSquare 

RSquare  Adj 

MAPE 

MAE 

-2  L  o  g  L  i  ke  I  i  hood 


3399.01513 
0.85850298 
0.85397507 
4.36744  977 
80088.1302 
3374.67746 


Figure  19.  R2,  MAD  (MAE),  and  MAPE  for  ARIMA  (l,l,l)x(l, 0,1)12 


Parameter  Estimates 


Term 

Factor 

Lag 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Constant 

AR1.1 

1 

1 

-0.067 

0.0137489 

-4.90 

<ooor 

Estimate 

AR2S12 

2 

12 

1.000 

8.1387e-7 

1.2e+6 

<ooor 

-0.0041645 

MA1.1 

1 

1 

0.606 

0.0666714 

9.10 

<.ooor 

MA2.12 

2 

12 

0.999 

0.0003128 

3192.4 

<ooor 

Intercept 

1 

0 

-8060.197 

12898.24 

-0.62 

0.5332 

Figure  20.  Parameter  estimates  for  p,q,P,Q  for  ARIMA  (1,1,1  )x(  1,0,1)12 


Figure  21.  Forecast  graph  for  ARIMA(l,l,l)x(l,0,l)12 
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d. 


DES 


We  used  JMP  to  formulate  appropriate  coefficients  for  our  DES  model.  Figures 
22  through  24  illustrate  the  R2,  MAD,  and  MAPE  of  this  DES  model  as  well  as  parameter 
value  estimates,  and  show  a  graph  of  the  model’s  outputs. 


Model  Summary 

DF 

128  Stable 

Yes 

Sum  of  Squared  Errors 

2.9566e+12  Invertible 

Yes 

Variance  Estimate 

2 .3099e+10 

Standard  Deviation 

151982.355 

Aka  i  Ice's  TV  1  nf  o  rm  ati  o  n  C  rite  ri  o  n 

3451.38877 

Schwarz's  Bayesian  Criterion 

3454.24858 

RSquare 

0.70533975 

RSquare  Adj 

0.70533975 

MAPE 

7.03647011 

MAE 

125778.062 

-2LogLikelihood 

3449.38877 

Figure  22.  R2,  MAD  (MAE),  and  MAPE  for  DES 


Parameter  Estimates 

Term  Estimate  Std  Error  t  Ratio  Prob=*|t| 

Level  Smoothing  Weight  0.11940809  0.0334009  3.57  0.0005* 


Figure  23.  Smoothing  constant  estimate  for  DES 


Forecast 


Figure  24.  Forecast  graph  for  DES 
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e. 


MA 


We  used  a  MA  model  to  compare  against  our  more  complex  models.  In  order  to 
predict  out  far  enough,  we  used  31  MA  coefficients.  Figures  25  through  27  illustrate  the 
R2,  MAD,  and  MAPE  of  this  MA  model  as  well  as  parameter  value  estimates,  and  show  a 
graph  of  the  model’s  outputs. 


Model  Summary 

DF 

99  Stable  Yes 

Sum  of  Squared  Errors 

1.2568e+12  Invertible  No 

Variance  Estimate 

1 .2695e+10 

Standard  Deviation 

112672.822 

Aka  i  ke's  “A  1  nf  o  rm  ati  o  n  C  rite  rion 

3509.65464 

Schwa rz's  Bayesian  Criterion 

3601.66095 

RSquare 

0.78851646 

RSquare  Adj 

0.7222  9434 

MAPE 

5.44399009 

MAE 

99667.8892 

-2LogLikelihood 

3445.65464 

Figure  25.  R2,  MAD  (MAE),  and  MAPE  for  MA 
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Parameter  Estimates 

Constant 


Term 

Lag 

Estimate 

Std  Error 

t  Ratio 

Probp|t|  Estimate 

MAI 

1 

-0.3978894 

0.1233023 

-3.09 

0.0026*  1799375.31 

MA2 

2 

-0.4747813 

0.111539 

-4.25 

<.0001* 

MA3 

3 

-0.3533395 

0.0393333 

-3.93 

0.0002* 

MA4 

4 

-0.296745 

0.0922315 

-3.22 

0.0017* 

MAS 

5 

-0.7791303 

0.1293497 

-6.02 

<.0001* 

MA6 

6 

-0.2233778 

0.1314175 

-1.70 

0.0923 

MA7 

7 

-0.4205171 

0.204092 

-2.06 

0.0420* 

MAS 

8 

0.01124107 

0.023567 3 

0.39 

0.6943 

MA9 

9 

-0.5880831 

0.1632436 

-3.60 

0.0005* 

MAI  0 

10 

0.11056503 

0.09736 

1.13 

0.2613 

MA11 

11 

-0.5259102 

0.1309615 

-2.91 

0.0045* 

MAI  2 

12 

-0.5945874 

0.1330624 

-4.47 

<.0001* 

MAI  3 

13 

-0.8109793 

0.1397731 

-4.27 

<  0001* 

MAI  4 

14 

-0.7466476 

0.152729 

-4.39 

<.0001* 

MAI  5 

15 

-0.597989 

0.2454064 

-2,44 

0.0166* 

MAI  6 

16 

-0.547467 

0.1439805 

-3.30 

0.0002* 

MAI  7 

17 

-0.8241105 

0.1643952 

-5.01 

<.0001* 

MAI  8 

18 

-0.948677 

0.1 

-9,47 

<.0001* 

MAI  9 

19 

-0.1826215 

0.1316925 

-1.39 

0.1636 

MA20 

20 

-0.4970215 

0.210137 

-2.36 

0.0200* 

MA21 

21 

0.03347745 

0.03414 

0.40 

0.6916 

MA22 

22 

-0,4582956 

0.244617 

-1.37 

0.0639 

MA23 

23 

-0.0542514 

0.1162133 

-0.47 

0.6416 

MA24 

24 

-0.6035083 

0.2654492 

-2.27 

0.0252* 

MA25 

25 

-0.2333143 

0.1415312 

-1.63 

0.0954 

MA26 

26 

-0.6633635 

0.2167167 

-3.09 

0.0026* 

MA27 

27 

-0.3331207 

0.1762033 

-1.92 

0.0579 

MA28 

28 

-0.1253301 

0.1154751 

-1.09 

0.2302 

MA29 

29 

-0.9409233 

0.1362121 

-5.05 

<.0001* 

MA30 

30 

-0.4372291 

0.1239651 

-3.53 

0.0006* 

MA31 

31 

-0.4320726 

0.1532661 

-3.05 

0.0030* 

Intercept 

0 

1799876 

117077.1 

15.37 

<.0001* 

Figure  26.  Parameter  estimates  for  MA 
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Figure  27.  Forecast  graph  for  MA 


/.  Operation  and  Maintenance  Budget  and  Workload  Analysis 

(1)  Description  of  O&M  Analysis 

We  analyzed  O&M  budget  requests  to  determine  if  they  could  be  used  as  a 
leading  indicator  to  predict  DLA  distribution  workload.  DLA’s  primary  customers  are  the 
Army,  Navy,  Air  Force,  and  Marines.  The  services’  “buying  power”  can  be  measured  in 
terms  of  annual  budgeted  authorization.  We  presumed  the  amount  DOD  requested  in 
O&M  funding  may  reflect  a  change,  either  increase  or  decrease,  in  operational  activity 
based  on  anticipated  mission  requirements.  An  example  would  be  if  the  Air  Force 
estimated  an  increase  in  flight  hours  for  the  next  FY  and  request  funding  accordingly.  An 
increase  or  decrease  in  flight  hours  results  in  logistical  requirements  that  are  then 
translated  into  additional  or  reduced  workload  for  DLA  Distribution. 

We  compiled  O&M  budgeting  information  from  the  Office  of  the  Under 
Secretary  of  Defense  Comptroller  website  for  FY2006  to  FY2013.  We  used  the  same 
aggregated  FY  issues  and  receipts  provided  by  DORRA  and  used  with  our  other  model 
analysis.  The  O&M  budget  request  contains  funding  for  training,  maintenance, 
administrative  costs,  and  purchases  from  DWCF  related  to  DLA  for  spare  parts.  The 
O&M  budget  includes  payments  to  support  allied  forces  and  multiple  other  expenses  that 
do  not  necessarily  predict  future  operational  activity.  We  included  additional  funding 
requests  for  overseas  contingency  operations  (OCO)  or  for  the  Global  War  on  Terrorism 
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not  included  in  the  base  budget  request.  We  further  narrowed  our  analysis  to  a  single 
budget  activity,  Operating  Forces,  which  we  believe  would  be  the  most  relevant  indicator 
to  change  in  the  DOD’s  “buying  power”  and  demand  on  DLA  Distribution. 

(2)  Results 

Over  the  past  eight  years,  O&M  funding  requests  have  increased  and  decreased 
due  to  changing  operational  needs.  DLA’s  workload  did  not  reflect  a  reaction  to  the 
change  in  funding  (Figure  28). 
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Figure  28.  Comparing  O&M  budget  requests  against  FY  total  issues  and 

receipts 


The  trend  lines  in  Figure  28  show  increases  in  O&M,  which  do  not  reflect  an 
increase  in  DLA  workload.  We  further  attempted  to  test  the  data  by  conducting  a  simple 
linear  regression  and  trying  to  predict  the  future  FY  DLA  workload  (Figure  29). 
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O&M  Request  vs.  Workload  [I&R1  >-n.o3S3n««i? 

M  *  3  R*  =  0.0732 
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O&M  Budget  Activity  "Operatng  Forces"  Funding.  Requests  in  2013  Dollars 

Figure  29.  O&M  vs.  I&R  regression 

The  resulting  model’s  R2  was  .0732.  The  model  predicted  FY2013  workload  of 
20,812,990,  which  resulted  in  a  3,799,289  MAD  (over  prediction),  and  a  22.3%  MAPE. 

(3)  O&M  Conclusion 

The  analysis  showed  O&M  budget  requests  do  not  provide  a  leading  indicator  to 
predict  changes  in  DLA  Distribution  workload.  In  fact,  DLA  Distribution  workload 
continued  to  show  a  downward  trend  without  regard  to  the  changing  O&M  budget 
requests. 

2.  Model  “Best-Fit”  Comparison  Based  on  All  Data 

The  model  comparison  matrix  in  Figure  30  shows  the  accuracy  and  precision 
criteria  used  to  determine  the  best -fit  model  for  our  workload  data  series. 
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MODEL 

R2 

MAD 

MAPE 

ARIMA  (2,1,2)  x  (2,0,1)12 

0.868 

77,084 

4.22% 

ARIMA  (1,1,1)  x  (1,0,1)12 

0.856 

80,088 

4.37% 

AREVIA  (0,1,1)  x  (0,1,1)12 

0.839 

76,859 

4.31% 

MA 

0.789 

99,668 

5.44% 

DES 

0.705 

125,778 

7.04% 

O&M  vs.  WL  Regression  Model 

0.073 

316,607 

22.30% 

DLA  Linear  Regression  Model 

0.019 

158,421 

13.31% 

Figure  30.  “Best  fit”  model  comparison  matrix 


From  these  results,  the  ARIMA  models  performed  best,  followed  by  DES  and 
MA.  Using  a  tlneshold  of  R2  =>  0.50,  we  remove  the  two  regression  models  from  further 
analysis. 

C.  FORECAST  COMPARISON 
1.  PBR  Cycle 

Figure  3 1  explains  the  fr  amework  of  how  we  conducted  workload  forecasting  for 
a  given  PBR  cycle  under  DLA’s  current  PBR  planning  timeline.  The  PBR  cycle  forecast 
encompasses  two  fiscal  years  of  monthly  data  points.  For  example,  PBR  16  forecast 
would  include  data  points  from  October  2014  until  September  2016.  The  input  data  for 
the  forecast  period  can  start  as  early  as  October  2003,  the  earliest  monthly  data  point,  and 
include  years  up  to  the  last  data  point  ending  on  time  period  “T.”  Time  period  “T” 
represents  the  February  prior  to  the  start  of  the  forecasted  PBR  cycle,  hi  this  example, 
time  period  “T”  would  be  February  2014.  Our  model  fit  analysis  focuses  on  the  years  of 
the  input  data  time  period,  and  our  back-casting  analysis  focuses  on  the  forecast 
predictions  for  a  given  PBR  cycle. 
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Forecasting  Framework 
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Figure  3 1 .  DLA  PBR  cycle  forecasting  framework 


2.  Back-Casting  Results 

a.  Forecasting  Error  Minimums,  Maximums,  Average  and  Range 

Figure  32  depicts  the  minimum  and  maximum  error  for  each  model.  The  best 
error  value  in  each  column  is  colored  green  and  the  worst  is  colored  red.  Interestingly,  the 
MA  model  produced  the  best  overall  minimum  error  in  the  first  year  but  its  maximum 
was  also  the  highest.  The  poorest  performing  model  was  determined  to  be  ARIMA 
(0,1, l)x(0,l, 1)12,  which  consistently  produced  the  highest  minimums  and  maximums. 
The  other  two  ARIMA  models  and  DES  were  determined  to  be  within  the  acceptable 
range  based  on  relative  comparable  results. 


Minimum  Error 

Maximum  Error 

1st  year 

2nd  year 

2  Yr  Average 

1st  year 

2nd  year 

2  Yr  Average 

ARIMA  (2,1,2)  x  (2,0,1)12 

243,558 

607,831 

425,694 

4,104.431 

6.109.895 

5.107J63 

ARIMA  (0,1,1)  i  (0,1,1)12 

1.513,232 

1,467,841 

1,529,266 

4,245,868 

8,417,344 

6,331,606 

ARIMA  (1,1, 1)  s  (1,0,1)12 

219,290 

263,421 

4,234,074 

6,314,583 

5,274,328 

DES 

390,419 

552,305 

4,531,780 

6,643,888 

5,587,834 

MA 

1,044,214 

569,564 

4,957,595 

6,001,266 

4,394,575 

Figure  32 .  Minimum  and  maximum  error  results  for  back-casting 


We  also  examined  the  average  errors  and  error  range  in  predicting  future 
workload.  The  MA  model  performed  worst  in  the  first  year  in  both  error  categories.  The 
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poorest  performing  model  was  determined  to  be  ARIMA  (0.1,l)x(0, 1.1)12,  which 
produced  the  highest  minimiims  and  maximmns.  Again,  the  other  two  ARIMA  models 
and  DES  were  determined  to  be  within  the  acceptable  range  based  on  relative  comparable 
results  (Figure  33). 


1st  year 

Average  Er 

2nd  year 

ror 

2  Yr  Average 

1st  year 

Error  Ran; 

2nd  year 

ge 

2  Yr  Average 

ARIMA  (2.1,2)  i  (2,0,1)12 

1,433,230 

2,354,067 

1,856,518 

3,860,873 

5,502,064 

4,681,469 

ARIMA  (0,1,1)  s  (0,1,1)12 

2,351,209 

5, 021,488 

3,702,875 

2,732.636 

6,949,503 

4,802,340 

ARIMA  (1,1,1)  i  (1,0,1)12 

1,315.624 

2,190,574 

1,745,092 

4,014,783 

6,051,161 

4,894,718 

DES 

1,441,219 

1,868,292 

I3M40 

4,141,361 

6,444.189 

5,035,529 

MA 

2,772,281 

2,904,519 

2,779,071 

4,862,681 

4,957.052 

Figure  33 .  Average  error  and  error  range  results  for  back-casting 


b.  Forecasting  Percent  Error  Minimums,  Maximums,  Average  and  Range 

Although  the  overall  errors  explain  the  total  workload  difference  for  each  year, 
we  used  the  percent  error  to  provide  a  better  estimate  to  compare  each  model.  The 
poorest  performing  model  was  again  determined  to  be  ARIMA  (0,1, l)x(0, 1,1)12,  which 
consistently  produced  the  highest  minimum  and  maximum  percent  error.  MA  performed 
best  for  the  minimum  percent  error  in  the  first  year  but  also  had  the  highest  maximum 
first-year  percent  error  (Figure  34). 


Minimum  PE 

Maximum  PE 

1st  year 

2nd  year 

2  Yr  Average 

1st  year 

2nd  year 

2  Yr  Average 

ARIMA  (2,1,2)  x  (2,0,1)12 

1.07% 

2.87% 

1.94% 

17,85% 

26.49% 

22.18% 

ARIMA  (0,1,1)  x  (0,1,1)12 

6.92% 

636% 

6.64% 

18.41% 

37.07% 

27.67% 

ARIMA  (1,1,1)  x  (1,0,1)12 

1.10% 

1.20% 

1  7% 

18.4% 

27.4% 

22.9% 

DES 

1  72% 

1.03% 

2.65% 

1971% 

28.81% 

24.26% 

MA 

0  41% 

4.60% 

2.51% 

26.25% 

35.27% 

24.81% 

Figure  34.  Minimum  and  maximum  percent  error  results  for  back-casting 
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ARIMA  (0,l,l)x(0, 1,1)12  and  MA  had  the  lowest  performance  when  assessing 
both  average  percent  error  and  percent  error  range.  The  other  two  ARIMA  models 
performed  the  best  while  DES  remained  within  close  relative  range  (Figure  35). 


Avera 

ge  Percentage  Error  (PE) 

PE  Ran* 

5* 

1st  year 

2nd  war 

2  Yr  Average 

1st  year 

2nd  year 

2  Yr  Average 

ARIMA  (2,1,2)  i  (2,0,1)12 

6.81% 

10.72% 

8.14% 

16.8% 

23.6% 

20.2% 

ARIMA  (0,1,1)  i  (0,1,1)12 

10.77% 

24.36% 

16.71% 

11.5% 

30.7% 

21.0% 

ARIMA  (1,1,1)  s  (1,0,1)12 

10.1% 

17% 

26% 

21% 

DES 

6.67% 

7.93% 

18.0% 

27.8% 

21.6% 

MA 

14.32% 

15.16% 

14.19% 

25.8% 

30.7% 

22.3% 

Figure  35.  Average  percent  error  and  percent  error  range  results  for  back- 

casting 


3.  Selection  Decision  (Model  Down  Select) 

We  selected  ARIMA  (2, l,2)x(2, 0,1)12,  ARIMA  (1,1, l)x(l, 0,1)12,  and  DES 
based  on  comparable  relative  performance  during  our  back-casting  analysis.  The  ARIMA 
(0,1, l)x(0,l, 1)12  total  error  and  percent  error  were  the  lowest-performing  results  and 
determined  to  be  excluded  from  further  analysis.  The  MA  model  performed  relatively 
well.  We  determined  to  exclude  MA  from  further  analysis  considering  its  large  average 
1st  year  total  and  percent  error,  and  the  largest  percent  error  range.  We  assessed  MA  to 
be  too  inconsistent  to  result  in  a  suitable  forecast  to  make  informed  business  decisions 
over  a  PBR  cycle. 

D.  SENSITIVITY  ANALYSIS 

A  sensitivity  analysis  enables  us  to  determine  how  reactive  our  remaining  models 
are  to  a  change  in  workload  (uptick  or  downtick).  The  actual  workload  experienced  at 
DLA  is  already  exhibiting  a  downtick,  so  we  use  that  to  measure  the  reacting  ability  of 
our  models.  We  simulate  an  uptick  to  measure  our  models’  reaction  to  a  future  change  as 
well. 
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1. 


Downtick  in  Actual  Data  from  FY10  to  FY13 


We  use  data  from  FY04  to  FY09  as  the  training  set  to  forecast  FY10  through 
FY13  (the  test  set)  for  ARIMA  (2,l,2)x(2,0,l)12,  ARIMA  (l,l,l)x(l, 0,1)12,  and  DES 
models  and  compare  with  the  actual  data  from  FY10  to  FY13.  We  then  add  FY10  and 
analyze  how  each  model  adjusts  to  the  change  in  workload  starting  in  FY10,  and  measure 
the  forecast  ability  for  FY11-FY13.  We  continue  adding  an  additional  year  to  the 
forecast  model,  measuring  the  accuracy  and  precision  of  each  model.  The  two-year  PBR 
cycle  of  the  test  set  is  measured. 

a.  Sensitivity  Analysis  Results 

The  downtick  scenario  results  are  depicted  in  Figures  36  through  38.  Since  part  of 
our  training  set  data  had  a  downward  trend,  our  models  will  not  have  a  large  percent  error 
compared  to  our  uptick  scenario,  which  simulated  a  significant  change  in  historical  trend. 
Each  graph  has  corresponding  tables,  which  both  represent  the  training  set  and  test  set 
summary  data  for  each  PBR  cycle.  The  green  box  highlighted  within  each  year  of  the 
PBR  cycle  represents  the  model  that  outperformed  the  other  models  based  on  percent 
error.  ARIMA  (1,1,1  )x(  1,0,1)12  displayed  better  results  for  the  majority  of  the  three 
downtick  PBR  cycles  examined.  No  model,  however,  displayed  results  outside  the 
relative  acceptable  range. 
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Figure  36.  Downtick  scenario  and  forecast  results  for  PBR  1 1 
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Figure  37.  Downtick  scenario  and  forecast  results  for  PBR  12 
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Figure  38.  Downtick  scenario  and  forecast  results  for  PBR  13 
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2.  Simulated  Uptick 

The  simulated  data  used  for  this  scenario  is  based  on  the  four  highest  absolute 
percent  changes  in  annual  workload,  which  are  8.4%,  9.6%,  10.3%,  and  12.4%.  Similar 
to  the  downtick  scenario,  we  incrementally  include  the  four  years  of  simulated  data  in  the 
forecast  and  measure  how  well  our  three  models  adjust  to  the  change  and  forecast  future 
values.  The  two-year  PBR  cycle  of  the  test  set  is  measured. 

a.  Uptick  Results 

The  uptick  scenario  results  have  a  larger  disparity  between  the  compared  models 
than  the  downtick  scenario.  The  difference  can  be  explained  by  the  trend  reversal 
simulated  in  the  uptick  scenario.  As  a  whole,  the  training  set  data  in  the  uptick  scenario 
exhibits  a  downward  trend  from  2004  to  the  start  of  the  simulated  data,  which  will  lead  to 
forecast  values  that  represent  the  same  downward  trend.  Therefore,  the  forecasted  values 
will  reflect  poor  error  values  and  will  not  represent  an  accurate  forecast  until  enough 
simulated  data  is  introduced  in  the  training  set. 

In  the  first  uptick  graph  (Figure  39),  when  only  5  months  of  simulated  data  is 
introduced  to  the  training  set,  ARIMA  (l,l,l)x(  1,0, 1)12  performs  better  than  the  other 
two  models  during  PBR  16.  These  results  could  conclude  ARIMA  (l,l,l)x(l, 0,1)12 
possesses  less  risk  during  an  initial  trend  change  in  workload,  but  the  difference  is  not 
significant  enough  to  support  that  claim. 
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Figure  39.  Uptick  scenario  and  forecasting  results  with  5  months  simulation 

data  introduced 


DES  performs  exceptionally  better  than  the  other  two  models  in  the  second  uptick 
graph,  PBR  17  (Figure  40),  which  has  17  months  of  simulated  data.  ARIMA 
(2, l,2)x(2, 0,1)12  performs  worst  during  that  same  time  period. 
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Figure  40.  Uptick  scenario  and  forecasting  results  with  17  months  simulation 

data  introduced 


During  PBR  18,  where  29  months  of  simulated  data  is  introduced  to  the  training 
set  data,  DES  performs  better,  but  the  remaining  models  are  not  far  behind.  ARIMA 
(2, l,2)x(2, 0,1)12  is  less  reactive  in  a  changing  environment,  while  DES  is  very  sensitive 
to  changing  training  set  data.  There  are  advantages  and  disadvantages  to  both,  but 
ARIMA  (1,1,1  )x(  1 ,0, 1)12  is  moderately  reactive  and  provides  less  risk  in  an  uncertain 
environment,  where  the  future  is  unknown  (Figure  41). 
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Figure  41 .  Uptick  scenario  and  forecasting  results  with  29  months  simulation 

data  introduced 


3.  Model  Results  Comparison 

Based  on  results  of  the  uptick  and  downtick  sensitivity  analysis,  we  select 
ARIMA  (1,1,1  )x(  1,0,1)12  as  the  final  model.  Although  DES  reacts  faster  to  the  uptick 
simulation,  ARIMA  (1,1,1  )x(  1 ,0, 1)12  has  a  more  tapered  reaction,  which  is  less  risky  in 
uncertain  environments. 

4.  Comparison  with  Current  Regression 

The  next  two  sections  compare  results  from  our  selected  model,  ARIMA 
(l,l,l)x(l, 0,1)12,  with  the  current  regression  over  two  PBR  cycles  and  translate  the 
workload  results  into  the  cost  required  to  fulfill  that  workload. 

a.  PBR  13  and  14  Comparison  (%  Error) 

To  see  how  well  our  selected  model  would  have  predicted  workload  if  it  were 
used,  we  compared  forecasts  between  our  model,  ARIMA  (1,1,1  )x(  1 ,0, 1)12,  and  the 
current  regression  model  DLA  uses.  We  used  PBR  13  and  PBR  14  since  we  had  actual 
data  to  compare  error  against.  We  replaced  actual  values  in  DLA’s  current  regression 
model  with  DORRA  numbers  to  keep  our  analysis  consistent  with  the  rest  of  our  study. 

58 


The  results,  as  shown  in  Figure  42,  show  that  ARIMA  (1,1,1  )x(  1 ,0, 1)12  does  a  better  job 
of  predicting  not  only  in  the  current  year  of  execution,  but  also  in  the  first  and  second 
years  of  the  PBR  cycle.  For  the  first  year  of  the  PBR  cycle  (defined  by  FY12  in  PBR  13 
and  FY13  in  PBR  14),  it  improved  the  percent  error  by  almost  42%.  The  current 
regression  model  had  an  average  percent  error  of  8.01%,  calculated  by  averaging  6.80% 
for  FY12  in  PBR  13  with  9.21%  for  FY13  in  PBR  14.  On  the  other  hand,  our  ARIMA 
(1,1,1  )x(  1 ,0, 1)12  model  had  an  average  percent  error  of  4.67%,  calculated  by  averaging 
1.13%  for  FY12  in  PBR  13  with  8.203%  for  FY13  in  PBR  14.  For  the  second  year  (FY13 
in  PBR  13),  it  improved  upon  the  current  regression  model  by  47%  from  a  17.47%  error 
with  the  current  regression  model  to  a  9.21%  error  with  the  ARIMA  (l,l,l)x(l, 0,1)12 
model. 
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Figure  42.  Comparison  of  ARIMA  (l,l,l)x(l, 0,1)12  to  DLA’s  current 
regression  model  in  PBR  13  and  PBR  14  forecasts 


b.  Translation  of  Workload  Forecasts  into  Cost  Allocations 

To  demonstrate  the  impact  of  workload  forecasts  on  DLA  Distribution,  we 
convert  the  forecasted  workload  into  a  cost  that  DLA  would  allocate  toward  fulfilling  the 
workload  requirements.  Using  the  current  lines  per  paid  equivalent  (LP/PE)  of  423  lines 
per  month  that  DLA  Distribution  uses,  we  can  estimate  the  average  number  of  FTEs  per 
month  needed  to  fulfill  the  workload  requirements.  From  this  FTE  number,  we  determine 
the  average  annual  cost  using  a  government  civilian  employee  average  cost  of 
$43. 07/hour  (U.S.  Department  of  Labor,  Bureau  of  Labor  Statistics,  2014)  multiplied  by 
the  standard  hours  per  year  for  government  employees  of  2,087  (Office  of  Personnel 
Management,  n.d.).  For  the  two  years  of  the  PBR  13  cycle,  we  calculate  the  cost  for  the 
actual  workload  experienced,  the  workload  using  the  current  regression  model  forecasted, 
and  the  workload  our  ARIMA  (l,l,l)x(l, 0,1)12  model  forecasted.  The  result  is  illustrated 
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ill  Figure  43.  If  our  ARIMA  (l,l,l)x(l,0,l)12  model  were  used  for  PBR13,  DLA 
Distribution  could  have  theoretically  saved  5.68%  ($19.5  million)  in  the  first  year  (FY12) 
and  8.26%  ($24.9  million)  in  the  second  year  (FY13). 
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Figure  43 .  PBR13  cost  comparison  of  current  regression  model  forecast  and 

ARIMA  (l,l,l)x(l,0,l)12  forecast 


We  used  the  same  process  to  predict  for  the  current  PBR  cycle,  PBR  16.  Figure 
44  displays  the  forecast  results  for  the  current  year  of  execution,  FY14,  and  the  first  two 
years  of  the  PBR  cycle,  FY15  and  FY16. 
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Figure  44.  PBR  16  forecast  comparison 
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For  FY15,  using  ARIMA  (l,l,l)x(l, 0,1)12  would  have  led  to  12.7%  ($36 
million)  less  in  costs  associated  with  the  predicted  workload  then  the  current  regression 
model.  For  FY16,  these  costs  would  have  been  20%  ($56  million)  less  (Figure  45). 
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Figure  45.  PBR16  cost  comparison  of  current  regression  model  forecast  and 

ARIMA  ( 1 , 1 , 1  )x(  1 ,0, 1 )  1 2  forecast 


E.  SUMMARY 

This  chapter  described  how  we  analyzed  the  data  series  we  gathered  and  our 
framework  for  forecast  modeling  and  model  comparison.  We  used  “best-fit” 
determination  to  select  models  to  analyze  further  for  predictive  characteristics.  We  then 
used  a  back-casting  technique  to  further  select  the  models  that  were  best  at  prediction. 
Lastly,  we  selected  our  recommended  model  based  on  sensitivity  analysis  using  an  uptick 
and  a  downtick  scenario.  Our  model  was  then  compared  to  the  current  regression  model 
to  illustrate  the  monetary  impact  of  how  our  model,  if  used,  would  have  predicted 
workload  over  recent  PBR  cycles.  The  next  and  final  chapter  of  this  study  provides  our 
conclusions  and  recommendations  to  DLA  as  well  as  recommendations  for  further 
academic  research  related  to  this  topic. 
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V.  SUMMARY,  CONCLUSIONS,  AND  RECOMMENDATIONS 


A.  SUMMARY 

This  project  analyzed  the  current  DLA  regression  model  used  to  estimate  DLA 
distribution  processing  workload.  The  distribution  workload  forecast  is  used  to  help 
properly  staff  distribution  centers  and  set  rates  to  fully  recover  costs.  DLA’s  current 
forecasting  method  is  not  sufficiently  predicting  workload.  We  examined  DLA's  current 
method  and  developed  multiple  forecasting  models  in  order  to  determine  a  more  accurate 
method  for  DLA  to  predict  workload. 

In  Chapter  II,  we  discussed  DLA's  background  and  current  regression  model  used 
to  forecast  its  total  workload.  We  identified  factors  and  issues  affecting  the  accuracy  in 
predicting  DLA  Distribution  workload  and  explained  our  understanding  of  the  current 
model.  We  discussed  previous  studies  on  forecasting  and  described  the  forecasting 
methods  we  used  in  detail. 

In  Chapter  III,  we  developed  a  methodology  for  our  analysis  and  model 
formulation  process  to  apply  against  the  forecasting  problem. 

In  Chapter  IV,  we  examined  the  results  for  each  model  against  their  fit  and 
accuracy  in  predicting  future  workload.  We  selected  a  final  model,  ARIMA 
(1,1,1  ,)x(  1 ,0, 1)12,  determined  to  be  the  best -performing  alternative  to  the  current  DLA 
method.  Finally,  we  compared  our  workload  model  predictions  against  DLA's  current 
process. 

B.  CONCLUSIONS 

Our  research  asked  two  primary  questions.  First,  what  are  the  shortcomings  of  the 
current  regression  model?  Second,  are  better  forecasting  methods  or  tools  available  to 
provide  more  accurate  forecasts? 

The  current  linear  regression  model  used  by  DLA  to  forecast  distribution  issues 
and  receipts  worked  relatively  well  as  business  grew  with  expansion  of  military 
operations  requiring  services  to  buy  more  items,  increasing  sales  and  seemingly 
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increasing  workload.  As  this  cycle  reversed,  the  model  showed  signs  of  weakness.  The 
relationship  between  sales  and  workload  may  not  be  as  significant  where  sales  is  the 
leading  indicator  causing  workload.  The  relationship,  symbolized  by  R2,  changes  from 
year  to  year,  indicating  that  there  may  be  other  external  factors  causing  workload. 
Additionally,  policy  and  supply  chain  accounts  change,  placing  additional  error  into  a 
forecast.  Related  to  this  is  the  fact  that  sales  are  also  forecasted  by  the  services  and 
supply  chains.  Using  forecasted  sales,  which  have  their  own  inherent  error,  to  predict 
workload  essentially  compounds  the  error  of  the  prediction. 

We  could  not  conclude  that  a  regression  model  based  on  sales  would  be  viable  in 
the  future.  A  separate  study  would  be  needed  on  how  sales  are  forecasted  and  on  the 
relationship  between  sales,  workload,  and  other  potential  factors  that  may  influence 
workload.  This  additional  analysis  was  beyond  the  scope  of  this  study. 

In  answering  the  second  and  principal  question  of  our  study,  we  conclude  that 
time  series  models  using  as  many  years  of  data  as  possible  are  viable  forecasting 
methodologies  for  predicting  issues  and  receipts.  ARIMA  and  DES  performed  well 
compared  to  the  current  regression.  Specifically,  an  ARIMA  (1,1,1  )x(  1 ,0, 1)12  performed 
the  best  out  of  the  models  we  tested  using  ten  years  of  data.  It  was  more  responsive  than 
an  ARIMA  (2,l,2)x(2,0,l)12  model  during  changes  and  forecasted  better  than  a  DES 
model,  which  was  over-responsive  to  changes.  It  was  able  to  predict  workload  for  two 
recent  PBR  cycles  (PBR  13  and  14)  more  accurately  than  DLA’s  current  regression 
model.  Our  forecast  error  for  the  first  and  second  years  was  4.67%  and  9.21%, 
respectively,  equivalent  to  a  42%  and  47%  improvement  over  the  current  model’s 
forecast  error. 

C.  RECOMMENDATIONS 

Recommendations  associated  with  this  study  are  for  DLA’s  operational  use  of  this 
model  and  future  forecasting  execution  as  well  as  future  academic  research  related  with 
this  topic. 
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1. 


Recommendations  for  DLA 


There  are  several  recommendations  for  DLA  to  incorporate  that  may  enable  a 
better  distribution  workload  forecast.  First,  we  recommend  that  DLA  use  ARIMA 
(1,1,1  )x(  1 ,0, 1)12  based  on  its  forecasting  performance  in  this  study.  Second,  reassessing 
the  model  periodically  will  help  identify  whether  the  current  model  is  still  the  best  model 
or  whether  a  change  is  needed.  Third,  we  recommend  examining  the  use  of  multiple 
models  in  combination  in  a  qualitative  manner  to  achieve  reliable  forecasts.  Periodic 
analytical  assessment  of  the  forecasting  model  is  critical  to  maintaining  accurate  forecasts 
and  relative  flexibility  to  adjust.  For  example,  the  DES  model  reacted  faster  to  drastic 
changes  (as  seen  in  the  Uptick  scenario),  so  an  option  may  be  to  use  both  DES  and 
ARIMA,  along  with  managerial  knowledge  and  judgment,  to  determine  a  more  accurate 
forecast  when  a  drastic  change  is  occurring  or  is  likely  to  occur. 

If  our  recommended  ARIMA  model  is  incorporated,  there  are  two  additional  ideas 
to  consider.  The  first  is  an  investment  in  a  forecasting  software  tool,  such  as  JMP,  to 
facilitate  running  the  model  and  analyzing  the  data  and  forecast  results.  A  trained  analyst 
using  the  software  tool  is  needed  to  examine  the  outputs  of  the  model  and  identify  areas 
of  present  or  future  variability  indicating  a  level  of  uncertainty  or  risk  in  the  forecast. 
Second,  we  recommend  running  the  model  more  frequently  (perhaps  quarterly)  in  order 
to  track  developing  trends  and  the  accuracy  of  the  forecasting  model.  Perhaps  the 
ARIMA  model  parameters  need  adjusting.  Perhaps  a  model  run  at  mid-year  confirms 
adjustments  that  are  planned. 

Additionally,  no  forecast  is  100%  accurate.  There  is  risk  that  is  tied  to  each  data 
point  being  predicted.  The  confidence  intervals  can  give  a  statistical  range  of  where  the 
value  is  likely  to  fall.  Understanding  how  the  model  behaves  can  give  decision  makers, 
based  on  their  operational  knowledge  of  past,  current,  and  future  activity,  an  idea  of  the 
level  of  this  risk  and  what  decisions  regarding  workload,  staffing,  and  spending  may  be 
required. 
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2. 


Recommendations  for  Further  Research 


While  conducting  our  research  and  analysis,  we  identified  a  number  of  areas  that 
we  felt  warranted  additional  study.  The  first  is  a  deeper  look  at  workload  cycles  at  DLA 
over  time,  or  perhaps  the  PBR  cycle  itself,  to  determine  if  there  is  an  ideal  time  period  to 
best  predict  a  full  PBR  cycle.  In  terms  of  this  study,  the  question  is  what  would  be  the 
best  time  period  to  use  in  the  training  set  to  predict  workload  in  the  PBR  cycle  of  the  test 
set. 

Another  recommendation  for  further  study  is  to  explore  the  use  of  a  regressor 
with  the  ARIMA  model.  If  sales  is  determined  to  be  a  causal  indicator  of  workload, 
adding  it  as  a  regressor  to  the  ARIMA  model  may  enable  the  model  to  predict  changes 
better  adjusting  with  the  signal  that  sales  presents. 

We  also  recommend  further  study  on  how  sales  are  forecast.  This  may  shed  light 
on  whether  it  is  a  useful  mechanism  for  predicting  issues  and  receipts  and  why  it  is  or  is 
not  useful.  As  part  of  this  or  perhaps  a  separate  study,  an  exploration  of  buying  power 
over  time  and  the  effects  that  may  have  on  the  relationship  between  sales  and  workload 
would  be  a  valid  study. 

Lastly,  scaling  the  workload  analysis  down  to  a  single  distribution  center  and 
studying  the  process  of  how  ground-level  workload  feeds  into  the  staffing  and  spending 
systems  may  also  be  feasible.  As  an  extension  of  this,  examining  how  workload  forecasts 
affect  the  pricing  and  staffing  decisions  for  DLA  Distribution  as  a  whole  would  be  a 
beneficial  continuation  of  workload  forecast  analysis  and  its  impacts. 

These  additional  topics  are  areas  beyond  the  scope  of  our  study,  but  they  would 
enhance  the  body  of  critical  analyses  of  DLA  systems,  provide  further  insight  and 
learning,  and  potentially  identify  efficiencies  that  DLA  could  implement  for  the  future. 
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Table  5.  Available  data,  in  aggregated  issues  and  receipts,  used  during  this 

study  and  broken  down  by  fiscal  year. 
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